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ABSTRACT 


There is a great opportunity for collaborative learning when agencies conduct emergency 
preparedness exercises together. If different members of the community contribute to the 
development of these exercises, then this learning benefits the entire population. As it 
stands, preparedness exercises are being conducted with minimal regard to 
recommendations from previous exercises and real-world events. Along with the 
incorporation of intelligence into these exercises, the objectives should promote a more 
inclusive design process based on focused relevance, encouraging agencies to view 
themselves more as members of the greater community rather than individual entities. 

Terrorist organizations learn from past failures as well as successes, and 
emergency responders should strive to parallel this learning in order to develop tactical 
improvements. Emergency responders need to promote the idea of intelligence-driven 
exercise design in order to support community resilience through collaborative training. 
Municipalities should spearhead this effort, supported financially by the private sector. 
With this fusion of intelligence and collaborative exercise design, we can leam from the 
fires of yesterday and prepare for the emergencies of tomorrow. 
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EXECUTIVE SUMMARY 


Emergency preparedness has changed with the times. It used to be that disasters were 
viewed as independent events with scenario-specific planning, response and recovery 
issues. 1 After September 11, terrorism became the focus of preparedness efforts, and 
later, following Hurricane Katrina in 2005, the focus shifted again to an all-hazards 
approach. 2 

Regardless of the nature of the emergency, collaboration among competing 
agencies is often a difficult proposal. Each agency has individualized needs, and its 
survival may depend upon satisfying these needs before fulfilling those outside its area of 
influence. In order to prepare a community to respond to and recover from any disaster, it 
is important that the community views itself as a whole. 

To define a community may be difficult. There are groups within each agency. 
Each municipal agency or private stakeholder is part of the greater community, and 
therefore, has a responsibility to participate in its collaborative preparedness. Fire and 
police departments may view themselves as independent groups, but they will work 
together with emergency medical personnel as members of a first responder collective. 
Agencies, public and private, may consider themselves part of a community that is not 
even aware of them, as may be the case with private security and local law enforcement. 3 
The members of each group must identify themselves as members of the community first 
and as members of individual factions second. In this way, they can contribute to the 
improvements of the collective and all groups reap the benefits. 

Agencies conduct preparedness exercises in an attempt to ready their personnel to 
respond appropriately in the event of an emergency. These exercises are evaluated and 
analyzed, and the results are provided to the agency in an effort to correct areas for 

1 Donald A. Donahue et al., “The All Needs Approach to Emergency Response,” Homeland Security 
Affairs 8, no. 1 (February 2012). 

2 Department ofFIomeland Security, Quadrennial Homeland Security Review Report (New York: 
Department ofFIomeland Security, February 2010). 

3 John Collicutt, “Community Resilience: The Future of Business Continuity,” Journal of Business 
Continuity & Emergency’ Planning 3, no. 2 (2009): 145-152. 
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improvement or sustain strengths. The purpose of exercise recommendations is to foster 
improvements. The same can be said for recommendations from real-world events. Issues 
identified during these should be incorporated into future exercises and into an agency’s 
corrective actions. 

In order to create a baseline to demonstrate the relationship between 
recommendations and exercise objectives, a study of After Action Reports (AAR) was 
conducted. Designed to compare and contrast recommendations and objectives from 
exercises and real-world emergencies, this study revealed a series of inconsistencies. 
Several recommendations appeared repeatedly and received minimal inclusion into 
exercise planning. This gap results in exercises not achieving their desired effect. Until 
the coordination between recommendation and objective is realized, exercise planners 
may continue to miss their mark, and exercises will fail to deliver the desired impact on 
emergency preparedness. 

The intelligence process is cyclically based. 4 Information is collected, analyzed 
then disseminated. This affects the consumer’s planning and needs and the result is 
evaluated. The feedback provided to the producers of intelligence influence the 
infonnation that is collected and the cycle continues. An exercise design program is 
likewise cyclical. Exercises are designed and conducted based on an agency’s needs. The 
resultant actions are evaluated and incorporated into planning for systematic 
improvement and future exercises. Corrective actions are implemented and reinserted 
into the design of future exercises. 

A strengthening of the exercise process can be anticipated if the two cycles 
intersect at the planning phase and intelligence is inserted into the exercise cycle. 
Exercise would then provide feedback to the intelligence analysts providing them with 
findings based on evaluation. This infonnation would allow the intelligence professionals 
to both look for information relevant for future exercises and analyze trends revealed 
during exercises that may support future intelligence paths. 


4 Keeley Townsend et al., “Intelligence-Led Mitigation,” Journal of Homeland Security and 
Emergency Management 7, no. 1 (2010). 
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The fusion of exercise design with intelligence creates a supportive relationship 
between both that improves the abilities of each. As each profession works toward its 
own goals and objectives with a common focus toward the needs of others as well, the 
entire process begins to produce a more effective product. This gives birth to the idea of 
Intelligent Exercise Theory (IET) and The Intelligent Exercise Model (IEM). As agencies 
embrace the IEM and begin to view preparedness exercises as a natural outreach of the 
intelligence process, it can be expected that communications between the intelligence and 
exercise communities will increase. Increased communications lead to increased patterns 
of interaction, coordination and decision-making. 5 Since a disaster is ultimately measured 
by a community’s ability to withstand its impact, this increase in coordinated abilities 
should strengthen the community’s resolve when faced with the future adversity. 6 

Ultimately, as a result the development of the IEM and adoption of Intelligent 
Exercise, the agencies of New York City can build a cooperative exercise design 
community made up of public and private stakeholders and supported by the 
municipality. Creating a platfonn where entities can benefit freely from each other’s 
exercises and intelligence analysis will encourage interagency collaboration and a 
common-goals approach to NYC preparedness. 


5 Clayton M. Christensen, The Innovator’s Dilemma: When New Technologies Cause Great Firms to 
Fail (Management of Innovation and Change), Reprint edition (Boston: Harvard Business Review Press, 
2013), 208. 

6 C. Lomnitz, “What Is a Disaster?” Natural Hazards 18, no. 1 (1998): 87-88. 
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I. INTRODUCTION 


In the midst of chaos, there is also opportunity. 

—Sun Tzu, The Art of War 


A. INTRODUCTION 

A population will be calmer in the aftennath of a terrorist attack or natural 
disaster if they know what to expect. 1 The municipality has a responsibility to train its 
employees on how to respond in the event of such a disaster. Through a robust exercise 
program, agencies can better prepare. Since 2004, the Homeland Security Exercise 
Evaluation Program (HSEEP) has offered guidance to agencies to conduct and develop 
exercises based on capabilities to prepare first responders and citizens for disasters, 
natural and man-made based on an area’s risk/vulnerability analysis. 2 By injecting 
intelligence into the risk/vulnerability analysis, a more refined and effective exercise 
program can be maintained contributing the greater improvement of an area’s community 
resilience. 

B. PROBLEM STATEMENT 

The Fire Department of the City of New York’s (FDNY) Center for Terrorism 
and Disaster Preparedness (CTDP) operates a Fire Service Intelligence branch as well as 
an Exercise Design program. These two entities provide the FDNY and other public and 
private city agencies with exceptional levels of exercise options, as well as critically 
astute analytical intelligence from a fire service perspective. The intelligence section 
weighs infonnation from around the world, from both classified and open sources, and 
provides relevant analysis to first responders. At the same time, the exercise design 
branch partners with various agencies throughout the city to create realistic exercise 
scenarios that serve the needs of the department as well as those of cooperating agencies. 

1 Consuela B. Pockett, United States and Israeli Homeland Security: A Comparative Analysis of 
Emergency Preparedness Efforts, The Counterproliferation Papers Future Warfare Series (Maxwell Air 
Force Base, AL: Counterproliferation Center, August 2005). 

2 “HSEEP-LLIS,” Department of Flomeland Security, accessed February 6, 2014, 
https://www.llis.dhs.gOv/.2004. 
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The cooperative fusion of these two branches exponentially increases the capabilities of 
each, resulting in a progressive, intelligence-led exercise design program. Ideally, 
intelligence should provide the exercise design team with relevant analysis of current 
threats facing city properties and personnel. This, in turn, should enable the exercise 
design team to develop practical scenarios based on realistic concerns. 

Unfortunately, intelligence and exercise professionals often operate independently 
of each other and often do not offer cooperative solutions to their department’s needs. 
The problem lies, not simply with a lack of coordination between intelligence and 
exercise design, but in the inability to overcome, what the 9/11 Commission referred to as 
“the human or systemic resistance to sharing information.” 3 This reluctance is not unique 
within the Department and can be seen on every level within the intelligence 
community. 4 To its credit, the CTDP intel section collaborates with the intelligence 
branches of other agencies, including the United States Coast Guard (USCG), Nassau 
County and NYC’s Office of Emergency Management (OEM), and the Nationwide 
Suspicious Activity Reporting (SAR) Initiative with the New York State Intelligence 
Center (NYSIC). Internally, the FDNY’s Bureau of Fire Investigation (BFI) is another 
partner. Yet there is a noticeable disconnect between two internal sections within the 
CTDP itself. In order to jointly fulfill the obligation to satisfy the needs of the department 
this gap needs to be addressed and minimized through changes to policy and culture. 

It should be noted that there is a systemic disconnect between the exercise design 
and intelligence communities. Limiting the involvement of intelligence in the design of 
exercise limits the breadth of the exercise and thus minimizes the lessons learned. The 
fire service historically uses intelligence to fight fires, especially those likely to occur 
again. Intelligence gathered today should also help mitigate the challenges of tomorrow. 

Exercise design is often based on the narrowly focused needs of individual 
stakeholders and facilities rather than on the more global needs of the community. When 

3 National Commission on Terrorist Attacks upon the United States, The 9/11 Commission Report: 
Final Report of the National Commission On Terrorist Attacks Upon the United States (Washington, DC: 
National Commission on Terrorist Attacks upon the United States, 2004). 

4 Richard A. Best Jr, Intelligence Information: Need-to-Know vs. Need-to-Share (Darby, PA: Diane 
Publishing, 2011). 
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agencies conduct parallel exercises separately, rather than exercising together, the 
opportunity to benefit from each other’s policies, procedures and lessons is lost. Since 
events and emergencies will undoubtedly involve multiple agencies, exercises should be 
conducted using as many agencies as appropriate. Objectives for these exercises must 
support the needs of each agency based on current intelligence and recommendations 
from previous events. The resultant chaos created by this collaborative gap, and the 
disconnect that exists between intelligence and exercise, creates an opportunity to 
strengthen the capabilities of each section. 

The development of a model that will regularly incorporate intelligence into 
exercise design planning will allow for consistent reevaluation of an agency’s needs and 
capabilities weighed against current vulnerabilities and threats. If intelligence is re¬ 
circulated into after action reports, then a steady succession of improvement can be 
developed. This model, then applied to an agency’s intelligence and exercise design 
sections, will serve as a guideline to integrate the two and enable a consistent stream of 
intelligence-led exercise design and intelligence heavy After Action Reports (AARs), 
culminating in recommendations based on experience. 

A 2010 Journal of Homeland Security and Emergency Management article 
explained the progression from Intelligence-led Policing to Intelligence-led Mitigation. 
Intelligent exercise is based on an extrapolation of Intelligence-led Mitigation. 5 

The premise of intelligence-led mitigation includes the intelligence-led 
policing approach for systematically collecting, organizing, analyzing, and 
utilizing intelligence to make informed resource decisions. However, 
while the goal of intelligence-led policing is to support law enforcement 
actions for crime prevention, disruption, and investigation, the goal of 
intelligence-led mitigation is to provide organizations with public safety 
and emergency service functions (including law enforcement) with 
intelligence products which enhance their understanding of the operational 
environment and enable them to make infonned resource decisions on 
appropriate preparedness, prevention, protection, response, and recovery 
actions to mitigate incidents. 6 


5 Keeley Townsend et al., “Intelligence-Led Mitigation,” Journal of Homeland Security and 
Emergency Management 7, no. 1 (2010). 

6 Ibid. 
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C. THESIS EXPECTATIONS 

The Intelligent Exercise Model (IEM) was created to demonstrate the cyclic 
benefits that can be developed when intelligence and exercise communities support each 
other. This thesis will explain the fusion of exercise and intelligence industries and how 
their collaboration will benefit both. Like intelligence-led mitigation, the idea of an 
intelligence-led exercise is to think proactively and use intelligence to enhance 
preparedness and minimize the impact of future disasters. 

The thesis will explore issues of organizational learning and seek to understand a 
lessons-leamed approach to improvement, despite agency biases and focus on the 
principles of motivation and human decision making during disasters. 7 As groups expand 
and view themselves as part of a larger group, e.g., the community, they are less 
organizationally protective than when they view themselves as only part of a much 
smaller group, e.g., an individual agency. It will analyze a broad representation of the 
intelligence and exercise communities affecting Homeland Security primarily associated 
with local municipalities. The thesis will then offer recommendations to fuse the interests 
of both the exercise design and intelligence communities on a broad scale. By presenting 
a mutually beneficial proposal, it is believed that Intelligent Exercise Theory will show 
how the intelligence and exercise cycles can operate in synchronicity. 

The purpose of this thesis is to address the greater intelligence and exercise design 
communities and expose an underlying inability to learn from one another and envision 
methods for operational collaboration between exercise and intelligence to operate in 
smarter collaboration. After Action Reports (AARs) from exercises and real-world events 
are being largely ignored as potential objectives in future exercises. This suggests that the 
emergency preparedness community is content to maintain the status quo rather than 
aggressively seek to improve community resilience. By developing and presenting the 
Intelligent Exercise Model (IEM) and Intelligent Exercise Theory (IET), this thesis will 
offer more positive and effective exercise design options. 


7 Donald A. Donahue et al., ‘‘The All Needs Approach to Emergency Response,” Homeland Security 
Affairs 8, no. 1 (February 2012). 
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II. LITERATURE REVIEW 


A. EXERCISE DESIGN 

Exercise can serve functions including training, gap analysis, planning, and team 
building. 8 As part of improvement planning, preparedness exercises can be used to 
identify areas where improvement is needed; to inform the process of improvements to 
the plans and procedures to enhance performance; and finally to re-exercise what 
evaluation has yielded as issues. In Emergency Management Exercises, 9 Regina Phelps, a 
consultant in the field of exercise since 1982, discusses exercises as a key to improving 
communications among disciplines. The book explores the planning of exercises and 
identifies the benefits of a sound exercise program. Exercises are conducted in order to 
validate training, “and practice strategic and tactical prevention, protection, response, and 
recovery capabilities in a risk-reduced environment.” 10 

Much of the exercise literature was gathered from over 100 After Action Reports 
(AARs), the AAR being the key evaluation deliverable, along with the accompanying 
improvement plan. The AAR should be viewed as both a science and an art 11 and should 
not be taken as insignificant. After-action reviews identify how to correct deficiencies, 
maintain the sustainability of our strengths, and focus on performance of specific training 
objectives. According to a May 2013 training document from the United States Fire 
Administration (USFA), AARs should answer the following 5 questions: 

• What was our mission? (Did we plan for this event? Were there any gaps 
in our planning?) 

• What went well? (Did we have adequate resources? Did we do all that we 
could?) 


8 Lisa R. Shugarman et at., Enhancing Public Health Preparedness (Santa Monica, CA: RAND, 
2005). 

0 Regina Phelps, Emergency’ Management Exercises: From Response to Recovery’: Everything You 
Need to Know to Design a Great Exercise (San Francisco, CA: Chandi Media, 2011). 

10 “About the Exercise & Training Program,” The Kansas Department of Health and Environment, 
accessed February 5, 2014, http://www.kdhe-exercises.org/. 

11 Margaret S. Salter and Gerald E. Klein, After Action Reviews: Current Observations and 
Recommendations (Arlington, VA: U.S. Army Research Institute for the Behavioral and Social Sciences, 
2007), http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA463410. 

5 



• What could have gone better? (Did we observe any unsafe behaviors? Was 
our training effective?) 

• What might we have done differently? (If we ran the same incident, what 
would we do differently?) 

• Who needs to know? (What needs to be fixed, and how do we pass this 
information to the correct source?) 72 

Several articles in the Journal of Business Continuity & Emergency Planning 
explore exercise potential both in the areas of development and evaluation. In order to 
build a tool that encourages documentation and analysis of relevant observations in 
exercises or real-world incidents to support decision makers in real time the evaluation 
process needs to be streamlined. This would enable the evaluators to facilitate the sharing 
of lessons learned, best practices and strategic improvements for the handling of future 
disasters and improve a “tool for collecting, processing and disseminating disaster-related 
infonnation, through its integration in a series of exercises.” 13 These exercises, which can 
be conducted to benefit multiple agencies as suggested by Margaret Crichton and Terence 
Kelly, need to be developed through a three-stage process of identifying the requirement 
for an exercise, obtaining approval for operations, and conducting an after-action review 
as part of a continuous improvement cycle. 14 While Crichton and Kelly address 
radioactive material transportation events, many of the situational factors can be 
generalized for the design of exercises in other areas with hazardous concerns. 

The uniqueness of health care-related exercises is explored in Making Acute 
Hospital Exercises More Realistic Without Impacting on Healthcare Deliver. Many times 
exercises are conducted in areas where service to the public cannot be interrupted, and 


12 “AAR: After Action Review in the Fire Service United States Fire Administration, accessed 
October 28, 2013, https://www.usfa.fema.gov/downloads/pdf/coffee-break/ms/ms_2013_3.pdf). 

13 Stewart D. Smith et at, “Disaster and Exercise Performance Information Collection Tool: Capturing 
Observations in Four Minutes or Less,” Journal of Business Continuity & Emergency Planning 6, no. 2 
(2013): 151-163. 

14 Margaret Crichton and Terence Kelly, “Developing Emergency Exercises for Flazardous Material 
Transportation: Process, Documents and templates,” .Journal of Business Continuity’ & Emergency 
Planning 6, no. 1 (2012): 32-46. 
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the benefits of exercise need to be carefully balanced with the organization’s 
responsibilities. 15 


1. Scenario Development 

The inherent strategy of exercise design is to develop an objectives-based 
scenario that allows the participants to validate their training in a realistic environment. It 
is important to consider the effect of scenario planning on exercise design. Scenario 
development gained popularity following World War II and was later used by 
corporations to speculate on economics, society and environmental concerns. 16 While the 
tenn “scenario planning” does not mean the same thing when applied to different 
strategies or organizations, 17 an acceptable definition of scenarios would include that 
they “are consistent and coherent descriptions of alternative hypothetical futures that 
reflect different perspectives on past, present, and future developments, which can serve 
as a basis for action. ” 18 

Scenario planning is not a prediction, despite the various meanings of the term. 19 
It is intended to help the strategizing organization determine whether they are adequately 
equipped to handle future challenges. 20 As we study the inclusion of scenario planning, it 
is important to look beyond one’s own industry and see how other factors affect 


15 Paul W. Riley, David J. Dalby, and Elizabeth A. Turner, “Making Acute Hospital Exercises More 
Realistic without Impacting on Healthcare Delivery,” Journal of Business Continuity & Emergency 
Planning 6, no. 2 (2013): 143-150. 

16 Philip van Notten, “Scenario Development: A Typology of Approaches,” in Think Scenarios, 
Rethink Education (Paris: OECD Publishing, 2006), 

http://www.oecd.org/site/schoolingfortomorrowknowledgebase/futuresthinking/scenarios/37246431.pdf. 

17 Kees van der Heijden, Scenarios: The Art of Strategic Conversation (Hoboken, NJ: John Wiley & 
Sons, 2005), 113. 

111 Philip van Notten, “Writing on the Wall: Scenario Development in Times of Discontinuity” (Boca 
Raton, FL: Dissertation.com, 2004). 

19 Van Notten, “Scenario Development.” 

20 Van der Heijden, Scenarios, 116. 
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expectations. 21 How individual agency capabilities are stressed and how a variety of 
“operational circumstances would stress those capabilities in very different ways” 22 

B. INTELLIGENCE SHARING 

An understanding of what constitutes intelligence is important before examining 
intelligence sharing. In Wanted: A Definition of Intelligence, Michael Warner asks, 
“Without a clear idea of what intelligence is, how can we develop a theory to explain 
how it works?” 23 Warner’s article includes definitions of intelligence dating back to the 
1950s. Most include words like clandestine, counterintelligence, foreign and secret. The 
FBI takes a broader, less cloak-and-dagger approach to an intelligence definition and 
describes it this way: “Intelligence is infonnation that has been analyzed and refined so 
that it is useful to policymakers in making decisions—specifically, decisions about 
potential threats. ” 24 These threats can be in any area of homeland security including the 
local level. Expanding upon that definition, the FBI website includes the following: 

• Intelligence is a product that consists of infonnation that has been refined 
to meet the needs of policymakers. 

• Intelligence is also a process through which that information is identified, 
collected, and analyzed. 

• And intelligence refers to both the individual organizations that shape raw 
data into a finished intelligence product for the benefit of decision makers 
and the larger community of these organizations 25 

Articles have addressed the desire and need to share intelligence. In a paper for 
the George Washington Law Review, Nathan Alexander Sales, assistant professor of law 
at the George Mason University School of Law, writes about intelligence sharing among 


21 Tony Kippenberger, “Pitfalls in Scenario Planning,” The Antidote 4, no. 4 (1999): 32-33. 

22 Paul K. Davis, “Uncertainty-Sensitive Planning,” in New Challenges, New Tools for Defense 
Decision making , ed. Stuart Johnson, Martin Libicki, and Gregory F. Treverton (Washington, DC: RAND, 
2003), 131-156, 

http://www.prgs.edu/content/dam/rand/pubs/monograph_reports/MR1576/MR1576.ch5.pdf. 

23 “Wanted: A Definition of‘Intelligence’ “ Central Intelligence Agency,” accessed February 4, 2014, 
https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-piiblications/csi- 
studies/studies/vol46no3/article02.html. 

24 “Intelligence Defined,” FBI, accessed February 9, 2014, https://www.fbi.gov/about- 
us/intelligence/defined. 

25 Ibid. 
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the intelligence community. He writes that intelligence guidelines in place often end up 
erecting a “wall” that discourages sharing. 26 He suggests that the wall “was not intended 
to be impregnable” and that there exists a quid pro quo whereby agencies could trade 
information on a mutually beneficial basis. 27 

In United States and Israeli Homeland Security: A Comparative Analysis of 
Emergency Preparedness Efforts, Consuela Pockett writes about the cooperation of 
intelligence resources within the Israeli security mission. If we examine intelligence 
sharing in other countries, the Israeli model stands out as one that provides citizens 
security through multi-layer cooperatives, where the military, security, and intelligence 
forces work together to supply an outer layer of protection through proactive operations 
against terrorist activities. 28 

A 2006 Homeland Security Affairs Journal article looks at the negative effects of 
not sharing intelligence. Authors Amy K. Donohue and Robert V. Tuohy ask the following 
questions regarding lessons not learned after a disaster. 

1. Is it true that lessons recur? 

2. What lessons are persistently identified? 

3. Why do these lessons continue to be identified as important? 

4. Why are these lessons so hard to leam? (That is, why do agencies have 
difficulty devising and implementing corrective actions once lessons are 
identified?) 

5. How do lessons-learned processes work? 

6. How can they be improved? 29 


26 Nathan Sales, “Share and Share Alike: Intelligence Agencies and Information Sharing,” George 
Mason Law & Economics Research Paper, no. 09-24 (2009): 287. 

27 Sales, “Share and Share Alike,” 288. 

28 Consuela B. Pockett, United States and Israeli Homeland Security: A Comparative Analysis of 
Emergency Preparedness Efforts. (Maxwell Air Force Base, AL: Counterproliferation Center, August 
2005). 

29 Amy K. Donahue and Robert V. Tuohy, “Lessons We Don’t Learn: A Study of the Lessons of 
Disasters, Why We Repeat Them, and How We Can Learn Them,” Homeland Security Affairs 2, no. 2 
(July 2006), http://calhoun.nps.edu/public/handle/10945/25094. 
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Donahue and Tuohy suggest that the intelligence gathered after a natural, 
accidental or man-made disaster is not being shared because “lessons are too often 
isolated and perishable, rather than generalized and institutionalized.” 30 

C. SOCIAL ISSUES AFFECTING DISASTER RESPONSE 

Enrico Quarantelli, Professor Emeritus and founding director of Disaster Research 
Center at the University of Delaware, is widely known as one of the founding scholars of 
the social science of disasters. His involvement in the area dates back to 1949 when he 
participated in the first systematic disaster field studies as a researcher in the National 
Opinion Research Center (NORC) team at University of Chicago. 31 Quarantelli studies 
the similarities in planning for natural and manmade disasters and notes that one method 
of defining a disaster is based on the notion that an event is only a “disaster” if the 
demands created by it exceed the community’s capacity for dealing with it. Quarantelli 
calls this an “imbalance in the demand-capability ratio in a crisis situation.” 32 Quarantelli 
implies that it is our level of preparedness that dictates whether or not a scenario becomes 
a disaster more than the mechanism of disaster. 

According to the 9/11 Commission report, in the aftennath of the September 11 
attacks, the FDNY was almost forced to accept the use of ICS, the scale of the response 
area was simply too large to manage any other way. 33 

The preparedness of the nation, post-9/11, was strongly focused on terrorism. In a 
post-Katrina world, the shift was toward planning for natural disasters. 34 The result is a 
realization that the source of a disaster is not nearly as important as a community’s 
capacity to prepare and respond. 


30 Ibid. 

31 “Faculty and Staff Bios,” University of Delaware, DRC-Disaster Research Center, accessed August 
29, 2013, http://www.udel.edu/DRC/aboutus/bios/Quarantelli.html. 

32 Enrico L. Quarantelli, What Is a Disaster? An Agent Specific or an All Disaster Spectrum Approach 
to Socio-behavioral Aspects of Earthquakes? (Newark, DE: Disaster Research Center, 1981), 453-78. 

33 National Commission, The 9/11 Commission Report. 

34 White House, The Federal Response to Hurricane Katrina: Lessons Learned (Washington, 
DC:White House, 2006), 12. 
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Historically, emergency management has been viewed as the responsibility of law 
enforcement and fire departments, with support in the event of a major catastrophe from 
public health and civil defense organizations. 35 Collaboration for homeland security can 
be defined as “agencies, organizations, and individuals from many tiers of public and 
private sectors, working, training, and exercising together for the common purpose of 
preventing terrorist threats to people and property.” 36 Public and private partnerships, as 
seen in the 2001 article in Disaster Prevention and Management titled “Disasters and 
communities: vulnerability, resilience and preparedness,” 37 may be a good indicator of 
the level of co-operation and assistance available within a community. This can provide a 
measure of the likelihood of the success of mitigation strategies that require collective 
and coordinated action being adopted and implemented. 38 Many of the papers reviewed 
here laud the benefits of cooperative efforts though few of these studies have emphasized 
the importance of the application of strategic management to emergency management. 39 

Several articles address social issues as affecting disaster response and 
management. Robert Kates, who helped establish the International Initiative for Science 
and Sustainability, notes that “the way men view the risks and opportunities of their 
uncertain environments plays a significant role in their decisions as to resource 
management.” 40 Essentially stating that an inaccurate assessment of risk will result in an 
inappropriate level of preparedness, Kates implies that the community needs awareness 
of risk before preparedness measures can begin. This opens the study of sociological 
differences between agencies and the willingness to recognize the benefit of training co¬ 
operatively. 41 David Alexander, professor and chief senior scientist at the Global Risk 


35 Sang Ok Choi, “Emergency Management: Implications from a Strategic Management Perspective,” 
Journal of Homeland Security and Emergency Management 5, no. 1 (2008): 2. 

36 William V. Pelfrey, “The Cycle of Preparedness: Establishing a Framework to Prepare for Terrorist 
Threats,” Journal of Homeland Security and Emergency Management 2, no. 1 (2005): 7. 

37 Douglas Paton and David Johnston, “Disasters and Communities: Vulnerability, Resilience and 
Preparedness,” Disaster Prevention and Management 10, no. 4 (2001): 270-277. 

38 Ibid., 274. 

39 Choi, “Emergency Managem,”l. 

40 R. W. Kates, “Hazard and Choice Perception in Flood Plain Management,” Research Paper No. 78, 
(Chicago: University of Chicago Press, 1962). 

41 Geoff O’Brien et al., “Approaching Disaster Management through Social Learning, ” Disaster 
Prevention and Management 19, no. 4 (2010): 498-508. 
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Forum in Davos Switzerland, 42 identifies six approaches to disaster research 
(geographical, anthropological, sociological, developmental, medical and technical) 
though he considers geography and sociology to be the most important. 43 He asserts that 
the geographical approach focuses on human-environment interactions, whereas the 
sociological approach has as its premise that “disasters are social events that reflect the 
ways, we live and structure our societies and communities.” 44 Simply stated, this idea 
leads into the study of the predictive nature of humans based on physical surroundings 
versus relationships. 

Literature addressing the sociological needs for and reluctance to exposing one’s 
agency to vulnerabilities 45 emphasizes the advantages of local jurisdictions conducting 
Threat, Risk, and Vulnerability Assessments (TRVAs) to aid in the effective allocation of 
resources. 46 Kerstin Eriksson identifies a reluctance to accept knowledge transfer 47 if the 
source is perceived as having originated outside a particular group or profession. In short, 
one group may be reluctant to openly share or receive knowledge with another simply 
because of organizational biases. 48 

In the area of knowledge transfer, Eriksson suggests that knowledge from 
previous experience, applied to a current situation, was of major significance in 
developing emergency response capacity 49 when researching her case study. She asks if a 


42 David E. Alexander, “Disaster Planning and Emergency Management" Disaster Planning and 
Emergency Management Blog, February 10, 2011, http://emergency- 
planning.blogspot.com/2011/02/offprints-available.html. 

43 David Alexander, Natural Disasters (London: UCL Press, 1993), 13. 

44 Ibid., 554. 

45 Allan Turner, Kenneth Glantz, and Gall, Julie Gall, “A Practitioner-Researcher Partnership to 
Develop and Deliver Operational Value of Threat, Risk and Vulnerability Assessment Training to Meet the 
Requirements of Emergency Responders,” Journal of Homeland Security’ and Emergency Management 9, 
no. 2, article 10. 

46 John Mueller. “Assessing Measures Designed to Protect the Homeland.” Policy Studies Journal 38, 
no. 1 (2010): 1-21. 

47 Kerstin Eriksson, “Knowledge Transfer between Preparedness and Emergency Response: A Case 
Study.” Disaster Prevention and Management 18, no. 2 (2009): 162-169. 

48 Joseph W. Pfeifer, “Command Resiliency: An Adaptive Response Strategy for Complex Incidents” 
(Master’s thesis, Naval Postgraduate School, 2005). 

49 Eriksson, “Knowledge Transfer,” 167. 
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well-designed analytic preparedness plan applied to effective knowledge transfer would 
have similar benefits. 

In an article about railway emergency preparedness, Gesine Hofinger, Robert 
Zinke and Stefan Strohschneider, psychologists from the Department of Intercultural 
Business Communication, Friedrich-Schiller-University in Jena, Germany, explain the 
psychological demands associated with disaster management. The trio studied the 
behavior of employees and civilians for the duration of a planned evacuation during a 
simulated below grade train accident. The impact of a quickly unfolding, dynamic event 
and the presence of the “fight or flight” tendency hinders the processes of conscious 
thinking and problem solving. Their analysis—’’weighing different options for action, 
asking critical questions—is nearly impossible while a strong tendency toward ad hocism 
can be observed,” 50 describes how emotion can lead to deviation from plans and should 
be taken into consideration during the formation of these plans. 

Thomas Drabek, in The Human Side of Disaster, explains the predictive nature of 
crowds and individuals during an evacuation. Drabek has written or co-written more than 
100 book chapters and journal articles and 27 books including The Human Side of 
Disaster (2009) and Human System Responses to Disaster: An Inventory of Sociological 
Findings (1986). He served as the co-editor of the International Journal of Mass 
Emergencies and Disasters (1986-1990) 51 Drabek points out that our society is more 
susceptible to disaster because of our developing technologies. 52 The items that create 
ease in our lives simultaneously create increased threats. 

D. ORGANIZATIONAL LEARNING 

Much has been written on the learning patterns of organizations, both positive and 
negative. In their two-part article titled Aptitude for Destruction, Brian Jackson et ah, 
discuss the learning and improvement plans of terrorist organizations and look at the 

50 Gesine Hofinger, Robert Zinke, and Stefan Strohschneider, “Role-specific Psychological 
Requirements in Preparing Public Transportation Staff for Disaster Response,” Disaster Prevention and 
Management 20, no. 4 (2011): 398-412. 

5 ' “Faculty and Staff Bios, University of Delaware, DRC-Disaster Research Center,” accessed August 
29, 2013, http://www.udel.edu/DRC/Alumni%20Bio%27s/Alumni%20Bio%20Pages/Drabek.html. 

52 Thomas E. Drabek, The Human Side of Disaster (Boca Raton, FL: CRC Press, 2013) 31. 

13 



variety of improvement strategies for different types of terrorist groups. A pattern of 
adapting and overcoming is necessary for groups to learn both from successes and 
failures. 53 It is offered in the article how anti-terrorist movements can be coordinated by 
understanding the ability to improve of these terrorist groups. The second part of the 
document looks specifically at five groups and their unique styles of organizational 
learning. It provides an in-depth explanation each group’s ideology, and how each has 
improved tactically and strategically or failed to leam from opportunities. 54 

Another way to look at how groups behave and leam can be found in the ‘writings 
of Martin Fishbein and leek Ajzenand and the theory of reasoned action. 55 The theory 
states that “behavior is rational and is dependent on the individual’s analysis of available 
information.” 56 However, in their article “A Conceptual Model of Gambling Behavior: 
Fishbein’s Theory of Reasoned Action,” Wm. Theodore Cummings and William Corney 
apply the theory to explain gambling patterns and behaviors; the research is applicable to 
organizational expectations. That is to say that an organizations learning is dependent on 
the infonnation available and obtained. This would be true in a simple or chaotic 
situation. These situations will likely cycle between complicated, complex and chaotic, 
with the restoration of simplistic as a goal. Decision makers will be expected to re¬ 
evaluate the situation throughout relying on the expertise and experience of personnel. 57 
Groups will analyze the infonnation available to them and select the behavior that is 
optimal. 58 Groups may also make decisions based on inconect information or a 
misinterpretation of it. Poker professional Annie Duke commented on decision making 


53 Brian A Jackson et at. Aptitude for Destruction: Organizational Learning in Terrorist Groups and 
Its Implications for Combating Terrorism Volume 1 (Santa Monica, CA: RAND, 2005), 
http://www.rand.org/publications/MG/MG331/. 

54 Brian A Jackson et al.. Aptitude for Destruction: Case Studies of Organizational Learning in Five 
Terrorist Groups, Volume 2 (Santa Monica, CA: RAND, 2005), 
http://www.rand.org/publications/MG/MG331/. 

55 Martin Fishbein and leek Ajzen, Predicting and Changing Behavior: The Reasoned Action 
Approach (New York: Psychology Press, 2010). 

56 Wm. Theodore Cummings and William Comey, “A Conceptual Model of Gambling Behavior: 
Fishbein’s Theory of Reasoned Action," The Journal of Gambling Behavior 3, no. 3 (Fall 1987): 190-201. 

57 David J. Snowden and Mary E. Boone, “A Leader’s Framework for Decision Making.” Harvard 
Business Review 85, no. 11 (November 2007): 68. 

58 Monte M. Page and Carolyn Wood Sherif, eds. Nebraska Symposium on Motivation!979: Beliefs, 
Attitudes, and Values (Lincoln, NE: University of Nebraska Press, 1980) 190. 
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saying, “Each decision should be made based only on the math at the decision-making 
point.” While it can be difficult to walk away from a project after committing extensive 
resources, often it is the wisest move. 59 

As organizations leam from outside influences, they should likewise be able to 
leam from each other. Jeanne B. Stinchcomb and Francisco Ordaz discuss the integration 
of lire and law enforcement agencies in an effort to streamline operations for the 
municipality, 60 understanding that successful corporations view organizational culture is 
as the “major distinguishing feature ... their most important competitive advantage, [and] 
the most powerful factor they all highlight as a key ingredient in their success.” 61 

It is important for organizations to benefit not only from successful endeavors, but 
to take advantage of the lessons learned when things do not go well. After a 2006 West 
Virginian mining disaster, Wyoming Senator Mike Enzi made the following statement: 
“We must draw upon the experience of these recent tragedies to find better ways to 
respond to mining accidents so that no other family is left without a father, a brother, or a 
son.” 62 A 2009 article in Organization Science addresses organizational learning by 
asking, “Do organizations learn vicariously from disasters experienced by others?” 63 
Suggesting that perhaps if a tragedy befalls one agency, only that agency will benefit 
from the lessons learned, where, ideally, peripheral agencies should take advantage of the 
tragedy to prevent similar disasters within their own arena. The article goes on to look at 
how organizations tend to “improve their perfonnance in a domain as they gain 
experience in that domain.” 64 


59 Annie Duke, “Sunk Cost,” Bluff Magazine, April 2007, http://www.bluff.com/magazine/sunk-cost- 
8962/. 

60 Jeanne B. Stinchcomb and Francisco Ordaz, “The Integration of Two ‘Brotherhoods’ into One 
Organizational Culture: A Psycho-Social Perspective on Merging Police and Fire Services,” Public 
Organization Review 7, no. 2 (January 30, 2007). 

61 Kim S. Cameron and Robert E. Quinn, Diagnosing and Changing Organizational Culture: Based 
on the Competing Values Framework, rev. ed (San Francisco: Jossey-Bass, 2006), 5. 

62 The State of Mine Safety and Health: Hearing of the Committee on Health, Education, Labor, and 
Pensions, United States Senate, 109th Cong., 2nd Session (2006). 

63 Peter M. Madsen, “These Lives Will Not Be Lost in Vain: Organizational Learning from Disaster in 
U.S. Coal Mining,” Organization Science 20, no. 5 (October 2009): 861-875, doi:10.1287/orsc.1080.0396. 

64 Ibid. 
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Peter M. Senge attributes this to a “Mental Models” all of us carry within 
ourselves. 65 Senge is credited with the iceberg model, which breaks “knowledge into 
three categories: events, pattern and structure.” 66 This is the basis for using scenarios as 
part of training. Not to predict actual events but to better understand the basis for 
expected actions. Scenario planners begin with the idea that there is much more to 
analysis than simply reporting and mimicking events as they occurred. 67 

Diane Vaughn of Boston College’s Department of Sociology, explores the 
negative side of organizations in The Dark Side of Organizations: Mistake, Misconduct, 
and Disaster. Vaughn suggests that organizations may perform detrimentally to positive 
expectations by confonning to organizational behaviors. 68 This may seem to run contrary 
to what Fishbein says regarding expected action, but may be just as predictable. Vaughn 
uses the tenn organizational deviance and defines it as “an event, activity, or 
circumstance, occurring in and/or produced by a fonnal organization that deviates from 
both formal design goals and normative standards or expectations, either in the fact of its 
occurrence or in its consequences, and produces suboptimal outcome. ” 69 

The psychological, organizational, social and cultural barriers to change can have 
a very powerful effect on whether a change model will be accepted or rejected and 
applied by the larger whole. 70 Tension develops any time new strategies or ideas that 
conflict with established culture. A group may find difficulties expanding its accepted 
sphere of influence because of group solidarity or its rejection of outsiders. This is 
consistent in what was presented by Henri Tajfel in his Social Identity writings. 71 The 
organization may view outsiders as a threat and seek to protect its position. It may 
struggle with differing opinions of needs internally as well. Because of “Structural 


65 Peter M. Senge, ‘‘Mental Models,” Strategy & Leadership 20, no. 2 (1992): 4-44. 

66 Peter M. Senge, The Fifth Discipline (New York: Random House Audio, 1994). 

67 Van der Heijden, Scenarios, 104. 

68 Diane Vaughan, “The Dark Side of Organizations: Mistake, Misconduct, and Disaster,” Annual 
Review of Sociology, 1999, 271-305. 

69 Ibid. 

70 Ibid. 

71 Henri Tajfel, Social Identity and Intergroup Relations (Cambridge: Cambridge University Press, 
2010 ). 
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secrecy” some at differing hierarchical levels within the organization may advocate for 
change while others resist. 72 Tension can be created when changes threaten to infiltrate 
an organization’s way of doing things, strategically, structurally or culturally. 73 

The four psychological barriers presented, “perception, homeostasis, confonnity 
and commitment, and personality factors” 74 are offered as issues that can affect an 
organization’s ability not only to accept potential new ideas and advance but its capacity 
to develop competitively with other organizations; a shortcoming that could result in the 
termination of an organization. Innovation is often a barrier in and of itself though there 
is often an advantage when innovation is developed from within an established agency. 75 
Creating a need for new technology, or in this case, a new system, is often met with 
skepticism. Organizations may be reluctant to accept a new system on top of a system 
they already understand and believe in. However, according to The Innovator’s Dilemma, 
if we can create a separate organization and embed it among potential new customers that 
need the new process, we can anticipate success as we move the innovation forward. 76 

E. COLLABORATION 

At the municipal level, collaboration in the areas of preparedness and disaster 
management is not only a practical use of resources but also a far more efficient 
application of them. Any municipality needs a strategy in order to prepare itself for 
emergencies. 77 


72 Diane Vaughan, The Challenger Launch Decision Risky’ Technology, Culture, and Deviance at 
NASA (Chicago: University of Chicago Press, 1996). 

73 “The Resistance to Change,” accessed February 4, 2014, 
https://sites.google.com/site/theresistancetochange/home. 

74 Ibid. 

75 Christensen, The Innovator’s Dilemma, 84-88. 

76 Ibid., 90. 

77 Steven Weiss, “Review of Communicating Emergency Preparedness: Strategies for Creating a 
Disaster Resilient Public ,” Journal of Homeland Security and Emergency’ Management 6, no 1, article 70. 
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Many documents address the failures associated with attacks of September 11, 
2001, as well as the response to Hurricane Katrina in 20 05. 78 The lessons learned have 
steered cooperative training toward a “whole-community” approach to preparedness. 79 
By learning from these events and involving multiple agencies in planning, we can ensure 
that redundancies best serve the greater good and help guide decision makers. 

There are a number of barricades related to inter-organizational collaboration. 
Some research indicates impracticality to combined training based on safety factors, cost 
and productivity. 80 These are often case by case and are rarely systemic. In Building 
Collaborative Capacity: An Innovative Strategy for Homeland Security Preparedness , 81 
the authors point to five organizational design components based on Jay Galbraith’s Star 


78 David M. Simpson, “Disaster Preparedness Measures: A Test Case Development and Application,” 
Disaster Prevention and Management 17, no. 5 (2008): 645-661. 

79 Donahue et al., “The All Needs Approach to Emergency Response.” 

80 Peter J. Smith, Ian Robertson, and Lyn Wakefield, “Developing Preparedness for Flexible Delivery 
of Training in Enterprises,” Journal of Workplace Learning 14, no. 6 (2002): 222-232. 

8 1 Susan Page Hocevar, Gail Farm Thomas, and Erik Jansen. “Building Collaborative Capacity: An 
Innovative Strategy for Flomeland Security Preparedness.” in Advances in Interdisciplinary Studies of Work 
Teams (Bingley, UK: Emerald, 2006), 255-274. 
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framework for organization design. 82 These are: purpose and strategy, structure, 
processes, incentives and people. These are then divided by success factors and barrier 
factors. For example, among incentives, leadership support and commitment is listed as a 
success factor while competition for resources is seen as a barrier factor. The star model 
depicts a support system that is key in collaboration; a support system that needs to exist, 
well before the idea of interagency preparedness efforts is explored. Dealing with hurdles 
in advance of their effect on planning will minimize their impact during the onset of the 
disaster. Kincaid suggests that there is a need for the training of personnel from different 
agencies, with different professions and standard operating procedures, to understand 
each other’s methods and procedures in order to coordinate response most efficiently. 83 
Some of this inter-agency training can be conducted using web-based techniques, 84 
which exposes each agency to the capabilities of the other, fostering a network of sharing 
as well as limiting the need for coordinating several agencies’ schedules. 

F. CONTENT AND THEMATIC ANALYSIS 

In order to develop a data series for this thesis, it became necessary to gain an 
understanding of a method to analyze data. Research in the areas of Content Analysis and 
Thematic Analysis shed some light on an important research technique. In Klaus 
Krippendorf s Content Analysis: An Introduction to its Methodology the author explains 
the method used to extract numeric value from text. Using an example from World War 
II where Allied analysts were able to extrapolate an accurate launch date for Germany’s 
V-weapon based on the monitoring and analysis of Joseph Goebbel’s speeches. 85 The 
example illustrates the effectiveness of the method prior to the widespread availability of 
computers for analytical purposes. 


82 Galbraith, Jay R. Designing Organizations: An Executive Briefing On Strategy, Structure, and 
Process (San Francisco: Jossey-Bass), 1995. 

83 J. Peter Kincaid, Joseph Donovan and Beth Pettitt, “Simulation Techniques for Training Emergency 
Response,” International Journal of Emergency Management J, no. 3 (2003). 

84 Jenvald, J., Morin, M. and J. Peter Kincaid, “A Framework for Web-based Dissemination of Models 
and Lessons Learned from Emergency-response Exercises and Operations,” International Journal of 
Emergency Management 1, no. 1 (2001): 82-94. 

85 Klaus Krippendorff, Content Analysis: An Introduction to Its Methodology (Beverly Hills: SAGE, 
1980), 17. 
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When analyzing data for interpretive results, there can be “considerable overlap 
among available qualitative approaches in terms of methods, procedures, and 
techniques.” 86 This is true in the area of participant observation which was a significant 
input to the methodology. As AARs are developed with contribution from participants, an 
added perspective is attached to the data. This, coupled with the contributions from 
“objective observers” leaving the analyst with additional interpretive challenges. 87 

As text is interpreted, its classification, now by computer, leads to a more accurate 
translation into a data set. Robert Philip Weber writes in Basic Content Analysis about the 
classifications of data, where the use of several classifications and sub-order 
classifications may lead to difficulties based on multiple meanings of the same text. 88 In 
choosing a methodology for this thesis that would allow understanding of the data 
presented, qualitative descriptive approaches such as content analysis and thematic 
analysis were chosen because they employ a “relatively low level of interpretation, in 
contrast to grounded theory or hermeneutic phenomenology, in which a higher level of 
interpretive complexity is required.” 89 

Both content analysis and thematic analysis are similar in that they analytically 
examine narrative materials from written recommendations by “breaking the text into 
relatively small units of content and submitting them to descriptive treatment,” 90 which 
fit the data analysis for this thesis perfectly. 

G. LITERATURE REVIEW CONCLUSION 

After a review of the literature associated with exercise design we are left to 
explain the connection between exercise design and societal biases preventing a 


86 Mojtaba Vaismoradi, Hannele Turunen, and Terese Bondas, “Content Analysis and Thematic 
Analysis: Implications for Conducting a Qualitative Descriptive Study: Qualitative Descriptive Study,” 
Nursing & Health Sciences 15, no. 3 (September 2013): 398-405, doi:10.1111/nhs.12048. 

87 Benjamin F. Crabtree and William L. Miller, Doing Qualitative Research (Thousand Oaks, CA: 
SAGE, 1999), 47. 

88 Robert Philip Weber, Basic Content Analysis (Newbury Park, CA: Sage Publications, 1990), 36. 

89 Vaismoradi, Turunen, and Bondas, “Content Analysis and Thematic Analysis.” 

90 Andrew Sparkes, “Narrative Analysis: Exploring the Whats and Hows of Personal Stories,” in 
Qualitative Research in Health Care, ed. Immy Holloway, 1st ed (Berkshire: Open University Press, 2005), 
191-208. 
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realization of the collaborative benefits. How one discipline within the community may 
or may not accept exercise design direction from the municipality. In this case, the 
FDNY, but under the proper circumstances any municipal agency may be capable of 
offering this assistance. The literature examines the social barriers that may hinder 
cooperation and prevent collaborative contributions to after action reporting. A further 
acceptance of differing thoughts and perspectives needs to be explored and some of these 
barriers identified and minimized. 

The literature looked at the different, predictable ways groups can be expected to 
behave during a disaster. This intelligence should also be applied to designing exercises 
for groups clustered for different reasons, for example, people grouped geographically in 
a train station or by virtue of similar employment like those working together in a high- 
rise office building. 

Much of the literature reviewed comes from scholarly articles in emergency 
journals. These articles relate the benefits learned from the experiences of the authors. 
Articles on behavior, predictability and organizational biases as well as a large sampling 
of AARs, culled for relevance. Further investigation of the sources used in the journal 
articles will apply increased credibility to the articles and thoughts expressed therein. 
There is a need for additional studies of emergency situations and examination of the 
results from earlier literature to extract relevant knowledge to be transferred into various 
preparedness activities and future exercises and response to crises. 91 

The USFA and U.S. Army’s infonnation on AARs lays out a framework for 
comprehensive AAR development, but moving forward, a greater emphasis needs to be 
directed toward the inclusion of intelligence. Intelligence that can lead to the 
development of a proactive rather than reactive exercise design process, one where we 
design exercises based on what we should prepare for rather than one where we look 
back on what we should have done. 


91 Kerstin Eriksson, “Knowledge Transfer between Preparedness and Emergency Response: A Case 
Study,” Disaster Prevention and Management 18, no. 2 (2009): 162-169, 


doi:10.1108/09653560910953234. 
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III. COLLABORATION AND RESISTANCE TO CHANGE 
IN DISASTER PREPAREDNESS 


It is a common defect in man to not make any provision in the calm 
against the tempest. 

—Niccolo Machiavelli, The Prince 


A. INTRODUCTION 

The greater Homeland Security mission of community resiliency can be better 
achieved through the collaborative efforts of the individual entities that make up the 
community. 92 This includes stakeholders both public and private. While this may seem a 
logical assumption, often private stakeholders do not take advantage of the opportunity to 
participate in training exercises organized and run by public agencies. Private 
stakeholders often do not even exercise with each other. Municipal agencies, fire, 
emergency medical, law enforcement, and public utilities rarely collaborate with each 
other, never mind with private entities. Effective collaboration requires both cultural 
sensitivity and a common language. 93 “Nonetheless, conflicts are inevitable, and some 
organizations simply may be unable or unwilling to work with others.” 94 The 
involvement of nongovernmental participants strengthens the community’s ability to deal 
with future disasters. The ability of a community to respond to and recover from disasters 
is not enhanced when officials preempt or exclude community involvement. 95 This 
chapter will examine organizational roadblocks and sociological limitations that prevent 
organizations from attaining a realistic measure of community resiliency, with the 
ultimate goal being increased willingness of all stakeholders contributing to the resiliency 
of the community. 


92 Hocevar, Thomas and Jansen, “Building Collaborative Capacity,” 255-274. 

93 William L. Waugh and Gregory Streib, “Collaboration and Leadership for Effective Emergency 
Management,” Public Administration Review 66, no. 1 (2006): 131-140. 

94 Ibid. 

95 Ibid. 
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B. 


WHAT IS A DISASTER? 


How does an event become a disaster? 96 Enrico Quarantelli, Professor Emeritus 
and founding director of Disaster Research Center at the University of Delaware is 
widely known as one of the founding scholars of the social science of disasters. 97 He 
detennines whether an event is a disaster by the preparedness level of the jurisdiction on 
the receiving end of the “disaster.” In their article “Response to Social Crisis and 
Disaster,” Quarantelli and Russell Dynes summarize basic substantive and structural 
trends that assume that a “disaster is primarily a social phenomenon and is thus 
identifiable in social terms.” 98 This is to suggest that the nature of the disaster, natural or 
man-made, is not as significant as the social impact when defining a disaster’s scope. 
David Alexander, professor and chief senior scientist at the Global Risk Forum in Davos 
Switzerland, 99 asserts that the localized approach focuses on human-environment 
interactions, whereas the sociological approach believes that “disasters are social events 
that reflect the ways, we live and structure our societies and communities.” 100 Simply 
stated, this idea leads into the study of the predictive nature of humans based on physical 
surroundings versus relationships. As responders, emergency planners need to focus less 
on the origin and more on the preparedness efforts of the community affected. 

Quarantelli’s argument is valid that “disasters are instabilities arising at the 
interface between society and the environment.” 101 Four factors must be considered when 
defining a disaster: “the physical agent, the physical consequences of the agent, the way 
in which the impact of the physical agent is evaluated, and the social disruption and 
social changes brought about by the physical agent and its impact.” “ Even where there 
is evidence of societal improvement driving community-based planning for disasters, this 
has arisen chiefly because of a lack of resources (the term disaster implies an 

96 Lomnitz, “What Is a Disaster,” 87-88. 

97 “Faculty and Staff Bios,” University of Delaware. 

98 E.L. Quarantelli and Russell R. Dynes, “Response to Social Crisis and Disaster,” Annual Review of 
Sociology 3 (1977): 23-49. 

99 Alexander, “Disaster Planning and Emergency Management.” 

100 “Editorial Board,” Journal of Geography & Natural Disasters, accessed August 8, 2013, 
http://www.omicsgroup.Org/journals/editorialboardJGND.php#. 

10 lomnitz, “What Is a Disaster,” 88. 

10 - Quarantelli and Dynes, “Response to Social Crisis and Disaster.” 
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103 

overwhelming of the community’s capacity to cope). That is to say that the disaster 
exists primarily due to a community’s inability to overcome the challenges presented by 
the event causing the “disaster.” If a tornado tears through a field and damages no 
infrastructure, can it really be called a disaster? 

In preparing for or responding to disaster, the local government has the first line 
of official public responsibility. However, the affected community will be responsible for 
itself in the initial minutes of any disaster until help arrives. Longer depending on the 
nature of impact and size of emergency management supports the local response effort by 
working with the public and private partners to develop emergency plans and capabilities 
that can respond to any natural or man-made hazard which threatens the community. 104 
We cannot think of natural disasters as being divine in origin or intent (“acts of God”) but 
recognize that whether natural or man-made the definition of the disaster is rooted in the 
“relationship between peoples and their environments.” 105 

C. LESSONS NOT LEARNED 

Lessons from disasters are often not incorporated into wider preparedness 
measures. Improvements are often only accepted as “doing it better” as opposed to 
“doing it differently.” 106 The thought behind a lessons-learned approach to evaluation is 
that we strive to improve. We believe that through practice we will continue to improve 
and better prepare ourselves and those for whom we are responsible. By conducting 
evaluations and summarizing findings, we share our knowledge across a lessons-learned 
format. Empirical data should then show some form of consistent improvement or at least 
a refocusing of efforts toward those areas where we are in the greatest need of 
improvement. However, in many cases, the opposite was true. According to the data 
collected, several recommendations appeared the most after real-world events and 


105 Geoff O’Brien et al., “Approaching Disaster Management through Social Learning,” Disaster 
Prevention and Management 19, no. 4 (2010): 498-508, doi: 10.1108/09653561011070402. 

104 Hocevar, Thomas, and Jansen, “Building Collaborative Capacity.” 

105 O’Brien et ah, “Approaching Disaster Management,” 499. 

106 Ibid. 
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exercises, namely: ICS concerns, communications, resources, EOC issues, and 
documentation. 

One could logically assume then that these would feature prominently as exercise 
objectives; however, as Table 1 shows, there was a tremendous disparity between 
recommendations and objectives. 


Table 1. Disparity between Recommendations and Objectives 


Recommendation 

Real World 

Exercise 

Objective 

% Dispa ritv 

ICS 

55% 

62% 

6% 

90% 

Communications 

70% 

34% 

6% 

82% 

Resources 

38% 

57% 

15% 

78% 

EOC 

48% 

62% 

15% 

39% 

Documentation 

45% 

38% 

15% 

61% 


ICS was included in recommendations from exercise AARs 62% of the time but 
only appeared as an objective in 6% of exercises. It was a recommendation in 55% of real 
world AARs. This indicates a gap between lessons learned and future planning If 
recommendations are made in either an exercise or real world AAR then there is a 
responsibility to incorporate it into training and eventually include it as an objective in 
future exercises in an attempt to validate training. By not addressing it, we are remaining 
complacent and expressing contentment with the status quo, despite the fact that the very 
system put into place to minimize complacency identifies the status quo as being 
ineffective. 

D. RESISTANCE TO CHANGE 

The need for collaboration often meets resistance as a result of social and 

organizational biases. That is, one group may limit their preparedness activities with 

another group based on preconceived notions. Also, one group may exhibit perceived 
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superiority to the other group based on intergroup relations. 107 Viewing the benefits of 
collaboration as weighted in favor of one side over the other creates a competitive search 
for value equity. FDNY Chief Joseph Pfeifer addressed the problems associated with 
group biases, stating that: 

Social identity that promotes power of one organization over another 
organization produces two social outcomes during complex incidents. 

First, there is the creation of a positive in-group bias toward those who are 
part of the same group and a negative out-group bias against those who are 
part of an alternate group. When providing information across groups, 
individuals are prone to give more infonnation to members of their own 
group and less to members outside that group. Secondly, when under 
stress, individuals feel little obligation to share valuable information with 
those outside their group since responsibility for acting is diffused within 
their in-group. This phenomenon excludes the out-group from receiving 
information that may be vital to their operation. 108 

This results in a siloed approach to preparedness, despite evidence available to 
individual organizations that collaboration would benefit each of them. So, why would 
organizations allow bias to interfere with safety? We have a tendency only to see the 
problems within our organizations that we are looking for. Therefore, additional 
perspectives applied to disaster response and preparedness are often unwelcomed and 
unutilized. When looking at an organization’s response plan for various disaster 
scenarios, the assumption is made that certain sequences of actions/events are likely to 
happen. 109 These expectations, and the assumed secondary and tertiary responses, are 
partly based in the practices and routines of our organizations and in our management 
strategies. 110 

Organizations fail at collaboration for many reasons: organizations have 
individual missions with goals and incentives that often conflict with one another; 
agencies often have histories of distrust that may be currently irrelevant but are, 


107 Tajfel, Social Identity and Intergroup Relations. 

108 Joseph W. Pfeifer, “Understanding How Organizational Bias Influenced First Responders at the 
World Trade Center” in Psychology of Terrorism ed. Brace Michael Bongar et al., (New York: Oxford 
University Press, 2007). 

109 Teemu Reiman and Carl Rollenhagen, “Human and Organizational Biases Affecting the 
Management of Safety,” Reliability’ Engineering & System Safety 96, no. 10 (October 2011): 1263-1274, 
doi:10.1016/j.ress.2011.05.010. 

110 Peter Wilkin, “The Ideology of Ergonomics,” Theoretical Issues in Ergonomics Science 11, no. 3 
(2010): 230-244. 
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nonetheless, hard to alter; leaders may not actively support collaborative efforts; 
and coordination systems and structures that might support collaboration are often 
lacking. 111 

There can be expected reluctance to try new ideas. Those aware of the 
improvements associated with a new idea face the challenge of “selling” this to a group 
that may only be conscious of the current practices. The Allegory of Plato’s Cave 
illustrates this well: 112 

Plato imagines a cave, in which several prisoners are tied in such a way 
that they cannot move or see anything but the stone wall in front of them. 

These prisoners have been in this position their entire lives. These 
prisoners have never seen outside of the cave. Behind them is a fire, and 
between them and the fire is a raised roadway. Other people walk along 
this walkway carrying shapes. When they do, the shadows of the shapes 
they are carrying are cast on to the wall. Having never seen the real 
objects before, the prisoners believe that the shadows of objects were 
“real.” 

Plato suggests that the prisoners would begin a “game” of guessing which 
shadow would appear next. If one of the prisoners were to correctly guess, 
the others would praise him as clever and say that he were a master of 
nature. 

Eventually, one of the prisoners escapes and leaves the cave. He is amazed 
at the world he discovers outside the cave and does not believe it can be 
real. He wanders about absorbing the sights and sounds of his new 
surroundings. As he becomes accustomed to his new world, he realizes 
that his fonner old world was inaccurate. 

He wonders at the sun and starts an intellectual journey consisting of these 
wonderful new experiences. He realizes also the game he had played with 
the other prisoners was now valueless. The prisoner returns to the cave, to 
share his amazing findings with the other prisoners. They do not believe 
him and threaten to kill him if he tries to free them. 113 


111 U.S. Government Accountability Office, Homeland Security: Management Challenges Facing 
Federal Leadership (Washington, DC:United States Government Accountability Office, December 2002). 

112 Plato, The Allegory’ of the Cave (Brea, CA: P & L, 2010). 

113 Ibid. 


28 




Figure 2. Depiction of the Cave 

Once the prisoner returns to the others in the cave, he tries to share his 
experiences but the group is not interested. They cannot imagine what he is telling them 
to be true and suggest he rejoin the group. It is not possible for him to forget what he has 
seen and he is forever influenced by his experience outside. Organizations react this way 
when faced with significant change. Those aware of the benefits of the new proposal 
instinctively wish to promote the idea while those unfamiliar experience reluctance to 
change. “That is not how we do things” is often the death knell to many innovative 
suggestions. 

The transition towards a new way of accomplishing goals takes time and 
should benefit from innovation and experience. Learning from events, we 
argue, should be a catalyst for thinking about how we do things going 
forward. Many published articles are narratives of a single disaster written 
from the perspective of one individual. 114 

In the UK, for example, the fire and rescue service has experienced a successful 
modification from “fire fighting” to “fire prevention” that exhibits the shift in emphasis 
of institutional thinking, from reactive (putting out fires) to proactive (preventing 
fires). 115 

114 Erik Auf der Heide, Disaster Response: Principles of Preparation and Coordination (St. Louis, 
MO: Mosby, 1989). 

115 O’Brien et al., “Approaching Disaster Management through Social Learning.” 
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Emergency management, as a profession has evolved into a more collaborative 
enterprise over the last 50 years. 116 There has been a gradual change away from the 
traditional hierarchical structure toward a more “dynamic and flexible network model 
that facilitates multi-organizational, intergovernmental, and intersectoral cooperation.” 117 

Professor Arthur Stinchcombe’s liabilities of newness theory explains 
comparative death rates of new and old organizations: A higher proportion of new 
organizations are likely to fail than old. Although Stinchcombe uses liabilities of newness 
to explain the development of populations of organizational fonns, his concept also may 
apply to analysis of cooperative movements, explaining why new programs or ideas 
struggle to succeed, or even the problems associated with transitioning new personnel 
into organizations. 118 

The Reiman and Rolenhagen table looks at what they describe as the four 
thematic areas of human and organizational behavior. These are beliefs about human 
behavior, beliefs about organizations, beliefs about infonnation and safety models. 119 
Reiman and Rolenhagen constructed the infonnation in Table 2 to display the integration 
of biases and their negative effect on the mission of resilience and safety demonstrating 
that biases should not be considered in isolation from each other—the biases are 
intenelated, as are the systems they address. The table can serve as a simplified depiction 
of how a bias in some area of safety management can have practical consequences for the 
way safety is managed in the organization. They suggest that these biases can lead to 
decision making in a vacuum which breeds complacency. There is a false sense of safety 
that develops from the deficiency of challenges to our safety; that a lack of catastrophe 
can be viewed as a lack of risk. 


116 Waugh and Streib, “Collaboration and Leadership for Effective Emergency Management.” 

117 Ibid. 

118 Vaughan, “The Dark Side of Organizations,” 275. 

119 Reiman and Rollenhagen, “Human and Organizational Biases Affecting the Management of 
Safety.” 
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Table 2. Consequences for Safety of the Biases in the Four Thematic Areas 
Concerning the Different Elements of Safety Management 120 


SAFETY 

MANAGEMENT 

Human 

behavior 

Organizational 

behavior 

Information and 
uncertainty 

Safety 

models 

Safety Policy 

BIAS OF REASONING: 

Expressed safety policies 
focus on most acute things 
since these are most 
salient, with “chronic” 
issues under-emphasized. 

CAUSAL PRIMACY 
OF STRUCTURE: 

Overreliance on 
technological means 
for 

controlling 

organizational 

performance 

UNCERTAINTY: 

The policy may 
overemphasize the need 
for confidence and 
certainty in an 
environment where total 
certainty is impossible 

SAFETY AS ABSENCE: 

The policy may 
overemphasize error 
prevention instead of safety 
promotion 

Organizing 

HUMAN ERROR BIAS: 

The starting point of 
organizing is in preventing 
errors and not in creating 
the context for safety 
performance 

CAUSAL PRIMACY 
OF STRUCTURE: 

Overreliance on formal 

organizational 

structure 

in controlling safety 

UNCERTAINTY: 

Uncertainty management 
may strive for reduction 
of uncertainty by 
standardization and 
formalization without 
perceiving the need for 
flexibility and 
adaptability. 

SAFETY AS ABSENCE: 
Relying on removing the 
negative may not reveal the 
underlying dynamics of the 
system that are essential for 
creating safety 

Risk analysis 

HUMAN ERROR BIAS: 

Human influence on safety 
is considered in negative 
terms only as potential 
errors that 

threaten the otherwise safe 
system 

CAUSAL PRIMACY 
OF STRUCTURE: 

Organizational 
phenomena such as 
norms and practices 
are not considered 
when evaluating the 
safety of 
the organization 

UNCERTAINTY: The 
estimatedprobabilities 
become treated like 
objective attributes of the 
environment due to 
misunderstanding of 
uncertainty 

SAFETY AS ABSENCE: 
Risk analyses may treat the 
absence of failures as a 
proof of safety 

Experience 

Feedback 

ATTRIBUTION 

ERROR: Event 
investigations seek to 
identify who made what 
mistake and attribute 
blame, instead of 
identifying how the actors 
made sense of the situation 
back then 

NEGLECT OF 
EMERGENCE: 

Social phenomena, 
such as 

norms and climate, 
that can contribute to 
both similar and 
different events in 
future are 
ignored in analyses 

OVER¬ 

QUANTIFICATION: 

Only quantitative data is 
considered as valid for 
risk analysis, neglecting 
harder to measure human 
and organizational issues 

LINEAR CAUSALITY: 

Ca n lead to focus on either 
technical, human or 
organizational factors 
as having caused the event 
under investigation 

Continuous 

Development 

ATTRIBUTION 

ERROR: 

Development activities 
might be 

targeted at changing 
individual behavior instead 
of the wider 
organization due to this 
bias 

LEARNING AND 
CHANGE: A static 
view on organization 
ignores the gradual 
change and optimizing 
of practices taking 
place continuously in 
the organization. 

UNCERTAINTY: A 
quest for having all the 
“facts” before making 
decisions or 
improvements may lead 
to an organizational 
inability to take action 

UNDER¬ 

SPECIFICATION: 

Lack of specification on 
what type of safety (e.g., 
process or personnel) an 
improvement focuses on can 
lead to wrong conclusions 

Safety Indicators 

BIAS OF REASONING: 

Indicator data can be used 
to justify preconceptions 
and 

expectations that the 
analyst had before seeing 
the data, due 
to a confirmation bias 

CAUSAL PRIMACY 
OF 

STRUCTURE: 

Indicators focus on 
structural elements 
such as technical 
reliability, 

management system, 
or instructions, 
neglecting e.g., social 
issues 

OVER¬ 

QUANTIFICATION: 

Only numerical indicators 
are considered as valid, 
missing other information 
such as intuitive 
judgments, social norms 
and subjective worries. 

SAFETY AS ABSENCE: 
Indicators may be selected 
to 

measure negative 
outcomes(absence) instead 
of proactive 

safety activities (presence) 


120 Ibid. 


31 













E. 


DISASTER SPECULATION 


A 2003 Pentagon research project called the Policy Analysis Market (PAM) was 
billed as “betting on terrorism.” 121 While the idea was so unpopular that the project was 
quickly cancelled, the idea is not without its merit. In designing a speculative market that 
would assist in making more specific forecasts regarding terrorist attacks, one of the 
developers of PAM described it as “a research project to test a new forecasting institution 
being considered as an addition to existing intelligence institutions, to help combine 
individual insights into a consensus forecast.” 122 

The aim of PAM was not to predict terrorist attacks but more to focus on their 
impact on geopolitical issues and destabilizing effects throughout the world that could be 
influenced by terrorist activities. Speculative markets have always used intelligence to 
achieve higher than average profits. 123 This intelligence is the product of both 
information and subjective opinion. 124 In this case, the analytical opinion of one capable 
of understanding the evolution of terrorist groups leading into future events. 

PAM’s unpopularity came from the prospective profit available through terrorism 
speculation. The fear was a terrorism-dependent market that could be manipulated, to the 
nation’s detriment, by terrorists. The politically unpopular project seemed to cross “a 
moral boundary,” 125 and despite the favorable review it received by those knowledgeable 
of its true intentions, 126 the plan was finished. While the possibility of financial gain 
available from wagering on the actions of terrorists may appear repugnant, profit can take 
on many shapes. If the profit potential was presented as lives and property saved from the 


121 Robin D. Hanson, “Designing Real Terrorism Futures,” Public Choice 128, no. 1/2 (July 2006): 
257-274. 

122 Ibid. 

123 Eric Zitzewitz and Justin Wolfers, “Prediction Markets,” The Journal of Economic Perspectives 
18, no. 2 (Spring 2004): 107-126. 

124 Stephen Figlewski, “Subjective Information in a Betting Market,” Journal of Political Economy 
87, no. 1 (1979): 75-88. 

125 Hanson, “Designing Real Terrorism Futures.” 

126 Robin Hanson, The Informed Press Favored the Policy Analysis Market (Fairfax, VA: George 
Mason University, 2005) 

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.67.3283&rep=repl&type=pdf. 
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speculative analysis of potential terrorist activities, one could have anticipated a more 
positive reaction toward PAM. 

The inability to collaborate in the name of disaster preparedness has a long 
history. Agencies, corporations and individuals often feel safer going it alone or 
continuing to do what has already proven not to work simply because it has always been 
done that way. When faced with a chaotic situation, people often revert back to a comfort 
system that they are familiar with. Organizations need to prepare under simple or kn own 
conditions in order to be able to function effectively under complex or chaotic situations 
(See Fig 3). Peter Wilkin references this idea when he states “the relationship between 
the necessary (structural relations) and the contingent (the acts of agents as groups or 
individuals) disappears.” 127 In a desire to achieve true community resiliency, 
organizations must first see themselves as members of the community and as individual 
agencies or businesses secondly. Organizations need to avoid preconceived notions that 
will compromise their emergency management capabilities, the safety of employees and 
their overall contribution to a resilient community. 


complex 

(parti'aNy repeatable, 
use guidelines) 


complicated 

(repeatable, focus on 
best practice) 



chaotic 

fflor rcpcatobio, 
usQ principles) 


simple 

{repeatable, compftanco 
to rates required) 


Figure 3. The Cynefin Framework 


This chapter has attempted to look at various social aspects that affect a mindset 
toward or away from collaboration and experience contributions from outside sources. In 


127 Wilkin, “The Ideology of Ergonomics. 
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an effort to increase safety and preparedness, and, through collaborative efforts, minimize 
organization deviances for public and private stakeholders. 

Collaboration alone will not overcome the roadblocks. Valuable lessons may be 
missed because of the bias held for one’s own organization or community. 128 Agencies, 
public and private, need to continually develop cooperative relationships and prevent the 
dissolving of existing ones. Without nurturing, the benefits of collaboration wither. 


128 E nr i co l. Quarantelli, Human Resources and Organizational Behaviors in Community Disasters 
and Their Relationship to Planning (Newark, DE: Disaster Research Center, 1982). 
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IV. ANALYSIS OF THE AFTER ACTION REPORTS 


Practice does not make perfect. Only perfect practice makes perfect. 

—Vince Lombardi, 
American football player and coach 


A. METHODOLOGY 

This thesis requires an evaluation of current exercise design value and the 
development of the Intelligent Exercise Model. The culmination is a greater collaboration 
between fire service intelligence entities and the exercise design process. Specifically 
designed for the FDNY’s Center for Terrorism and Disaster Preparedness (CTDP), the 
model will serve as a visual mechanism to reinforce the need for intelligence 
contributions to the exercise design process applicable to any municipal agency. 

To support thesis research. After Action Reports (AAR) from numerous events, 
exercises and incidents were collected and analyzed. The purpose was to conduct a 
content analysis, coding for themes and significant language that allows us to track 
correlation between actual response issues and exercise issues. These AARs reviewed 
were taken from exercises and actual responses to emergencies from within the United 
States that represent a comprehensive sampling of the evaluative process and its effect on 
improvement planning. This thesis is not focused on proximity of exercises and events. 
Since the focus is on shared intelligence across boundaries, location was not considered 
to be a limiting factor. While it will be expected that agencies are more likely to benefit 
from lessons learned proximal to their area of responsibility, agencies have a 
responsibility to seek out intelligence gathered from exercises and events and apply that 
to their jurisdiction. 
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B. 


CONTENT ANALYSIS 


Richard Boyatzis stated in Transforming Qualitative Information: Thematic 
Analysis and Code Development 129 that the four stages in developing the ability to use 
thematic analysis are: 

• Sensing Themes—recognizing the codable moment 

• Doing it Reliably—recognizing the codable moment and encoding it 
consistently 

• Developing Codes 

• Interpreting the Infonnation and Themes in the Context of a theory or 

Conceptual Framework—contributing to the development of knowledge 130 

There are three distinct stages involved in the use of thematic analysis: Stage I, 
deciding on sampling and design issues; Stage II developing themes and a code and stage 
III is where we begin validating and using the code. 

Stage II is further divided into three different ways to develop a thematic code: 

• Theory driven 

• Prior data or prior research driven 

1 T 1 

• Inductive data driven 

In Stage I, the sample of 101 AARs was chosen to be representative of the AAR 
following the HSEEP framework. This would reveal trends that could be graphically 
illustrated for demonstration purposes. 

In Stage II, the themes were revealed as the data were collected. Initially the 
categories used for coding the recommendations started at zero, since categories were not 
predetennined. 

Content analysis can be defined as “a research technique for making replicable 

1 T9 

and valid inferences from data to their context.” “ The intent here was to synthesize the 

1-9 Richard E. Boyatzis, Transforming Qualitative Information: Thematic Analysis and Code 
Development (Thousand Oaks, CA: SAGE, 1998). 

130 Ibid., 11. 

131 Ibid., 29. 

1 32 Krippendorff, Content Analysis. 31. 
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recurring recommendations from AARs for real world events and compare them to 
objectives established for exercises. Recommendations from exercises were collected as 
well. This data was then analyzed and compared. The four key components of Content 
Analysis are measurement, indication, representation and interpretation. 

1. Measurement 

The measurement procedures of content analysis are based on frequency counts of 
semantically equivalent textual units such as words, word senses phrase or themes. 134 
The data collected represented the frequency with which certain words or phrases 
appeared among the recommendations in each AAR. These were coded and sorted into 
one of, what eventually became 104 categories. The categories were chosen simply 
because they existed among the recommendations. Only one of the real world AARs 
included a recommendation that addressed a Special Weapons and Tactics (SWAT) team 
and only two included recommendations concerning the Stafford Act. Both were included 
as part of the data set for two reasons. One was to maintain objectivity. It was important 
not to omit any entries not matter how seemingly insignificant they may have appeared in 
order to create an accurate sample. Secondly, while the SWAT reference may have 
appeared statistically insignificant initially, its comparative value when paired against 
recommendations from exercise AARs becomes more evident. It was impossible to tell 
this during the collection process so all AAR recommendations were included. 

2. Indication 

Indication refers to the unmeasured aspects of the text as inferred by the analyst. 
For example, if an AAR refereed repeatedly to a recommendation, it would only be 
counted where it belongs. Special inclusion for overstated recommendation words such as 
must, never or always or for understated words such as may, could or some were not 
given as the objectivity of the person writing the AAR was not viewed as important as the 
content. 135 One AAR included obvious bias toward the issues affecting the special needs 


133 Weber, Basic Content Analysis .70. 

134 Ibid., 72. 

135 Ibid., 75. 
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community and nearly every recommendation had special needs concerns, though the 
objectives of that exercise were not specifically geared toward a special needs population. 
The recommendations were interpreted and categorized as needed for “specialized 
equipment,” “transportation resiliency” and “social media,” but each recommendation 
was categorized based on its content and the “Special Needs” category received its single 
checkmark. 


3. Representation 

Often criticized for its language coding limitations, content analysis can have the 
appearance of maintaining certain linguistic biases. 136 Since the interpretation of data 
into coded analysis is the responsibility of the analyst, it is important to classify words or 
phrases by the references they make. That is, to ignore the subjective preferences of the 
creator of the AAR and instead concentrate on the material recommendations. The 
intended definition was drawn by the comprehensive AAR and the analyst’s familiarity 
with the process. Any recommendations that were misinterpreted from the intention of 
the AAR author(s) was unintentional though would have been consistent throughout the 
information gathering process for this thesis. To summarize, though there may be more 
valuable text analysis systems, some distinctions in language were not considered worth 
maintaining. 137 

4. Interpretation 

Each recommendation needed to be interpreted in order to find its needed 
location. The data did not require a high level of interpretive complexity. 138 Specifically, 
if a recommendation was made that included an inability to recognize the Incident 
Commander in the EOC, then that was included as a recommendation in the EOC, ICS 
and ICS identification categories. While this would leave some AARs being represented 
in more categories than may have seemed necessary, it ensured complete inclusion and 


136 

137 

138 


Krippendorff, Content Analysis, 43. 

Weber, Basic Content Analysis, 77. 

Vaismoradi, Turunen, and Bondas, “Content Analysis and Thematic Analysis,” 399. 

38 



also minimized interpretation bias that would’ve resulted in using less categories based 
on the analyst’s familiarity with the AAR process. 

C. THE AFTER ACTION REPORTS 

1. Selection of the AARs 

The AARs selected for review were picked to cover a variety of emergency 
circumstances, which would be relevant to as many scenarios as possible. For example, 
the issues that face communities and responders during a hurricane have similarities to 
those that occur after tornados or blackouts. The effects of these emergencies on the 
healthcare system are similar to those that can be expected during a medical surge for a 
variety of scenarios. Therefore, the goal is to evaluate a large enough sample of 
emergencies to include as many of the core capabilities as possible. Significant events 
were included that were easily recognizable (i.e., the Oklahoma City bombing, Hurricane 
Katrina or the School Shooting at Columbine). Then, by comparing observations 
uncovered during emergencies with objectives established during the exercise design 
process, either a clear communal understanding of lessons learned will emerge or a gap 
will surface indicating a need for greater collaboration between intelligence and exercise. 

In order to determine whether the hypothesis is correct, that we are not 
incorporating lessons learned from actual events to exercises conducted afterward, AARs 
were selected where a similar exercise occurred subsequent to the event. Wherever 
possible, data from an exercise that was conducted before the actual event would be 
included to provide a metric of beneficial learning. Events were chosen where the AAR 
was developed following the format set forth in the Homeland Security Exercise and 
Evaluation Program (HSEEP). The same was done for the exercise AARs. This would 
lend consistency to the sampling and make the extraction of recommendations and 
objectives 

It is believed that these exercises did not necessarily need to be restricted 
geographically. Because of the availability of lessons learned platforms, exercises 
conducted in California are available to benefit exercise planners in New York. Likewise, 
the lessons learned from a hurricane occurring on the North Carolina Coast should 
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provide benefit to responders and emergency managers in Texas. Therefore, 
inconsistencies noticed between the AARs compiled after events and exercises are being 
evaluated on a systemic basis. That is, whether exercises are conducted as a direct result 
of a local emergency (Tornado exercise conducted in Joplin, MO or a Hurricane exercise 
in NY City). A gap in this area should strengthen the need for increased collaborative 
intelligence sharing. As common lessons learned emerge during emergencies, we should, 
ideally, see these identified as objectives in future exercises. It is the hypothesis of this 
thesis, however, that objectives and lessons learned will be dissimilar. 

The sample size was drawn from exercises and events around the country 
adhering to the following guidelines: 

• Only AARs for events occurring within the United States were included 

• AARs had to follow the HSEEP fonnat - 

• The range of AARs was from 2001 through the present - The goal was to 
focus primarily on post 9/11 recommendation-heavy climate adhering to 
modern fonnats. 139 

• Approximate even distribution was maintained between exercises and real 
world events to maintain parity. (53 exercises and 48 real World events) 

Searches for AARs were conducted on the DHS Lessons Learned Infonnation 
Sharing (LLIS) website and The Homeland Security Digital Library (HSDL) and 
additional Google searches. The Google searches yielded several collections of AARs but 
one of the most valuable was on the WisconsinWatch.org 140 website. The website was 
produced by the Wisconsin Center for Investigative Journalism who requested AARs 
from counties across Wisconsin. The reports range from 2005-2011 and were written by 
a variety of agencies. 141 This accounts for the disproportionate number of Wisconsin 


139 The only exceptions were the 1999 Columbine School shooting and the 1995 Oklahoma City 
bombing because of their significance as landmark events in homeland security, even though they occurred 
before the notion of “Homeland Security” came into existence. They were included because of their history 
and because the reports available were deemed sufficiently formatted in such a way that recommendations 
could be easily extracted. While other pre-9/11 events could also be viewed as seminal, e.g., the FBI 
standoffs at Ruby Ridge and Waco, Hurricane Andrew or the crash of TWA Flight 800, the primary focus 
was to be post 9/11. 

14° “What They Learned in the Disaster: Agencies’ After-Action Reports,” WisconsinWatch.org, 
accessed January 14, 2014, http://www.wisconsinwatch.org/viz/what-they-learned-in-the-disaster-agencies- 
after-action-rcports/. 

141 Ibid. 
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AARs represented in the database. Whether some states exercise more than others, 
produce more consistent AARs after exercises or emergencies, or simply made their 
AARs more accessible was unclear. 

New York City and Washington, DC, were singled out as sole city representatives 
because of the September 11 attacks. Figure 4 shows the AAR contribution by state for 
both exercise and real world events. 



■ Exercises ■ Reel World Events 

Figure 4. Exercise Contribution by State 

Once found, the AARs were briefly skimmed, first for U.S. origin then HSEEP 
compliance. They were then filed simply as an exercise or real world event. It was 
important not to read the AARs at this point in order to maintain the objectivity of the 
sample. 


2. Coding of the AARs 

Content and Thematic Analysis allow the analyst to remain objective when 
collecting data. Either the infonnation occurs or it does not. Since the AARs are all 
historical documents, meaning the infonnation entered into them was done so before and 
separate from the purposes of this paper, there is no interaction between data submission 
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data collection which could bias the findings. Content analysis helps those evaluating to 
learn more about the data they are examining because it is systematic. 142 

The coding of the AARs was based on the frequency of common exercise and 
emergency issues, for example, ICS, communications, patient management, etc. The 
occurrences of common phrases will be gathered and tracked and trends among AARs 
will be included. The information was then assigned to categorical variables 
distinguishing among subject, timing and event and classified into finite number of 
categories. 143 

The recommendation section of each AAR was coded for themes in the areas of 
improvement across exercises and real world events. As a new recommendation was 
discovered a new category was created. This eventually resulted in 104 different 
categories The codification of the AARs supplied a comparative metric whereby the 
information contained in the recommendations could be assigned a numeric value and 
consistently measured against the remaining information either wholly or partially. 

3. Analysis of the AAR Coding 

The occurrence rate among emergency AARs was compared to the frequency of 
inclusion during exercises. The expectation is this: if the AAR following an emergency 
identifies a shortcoming, this gap should be addressed in future exercises. 

As emergencies occur across the nation, the frequency of issues raised exhibits 
consistency from region to region; therefore, the lessons learned should similarly 
maintain a consistency that is translated to exercise development regardless of location. 

The data were collated into over 100 classifications. The first category that was 
tracked was the amount of warning time provided to civilians and responders during the 
scenario, whether exercise or real. The scenarios were divided among: No warning, 
minimal warning and significant warning events. No warning events were those where no 
realistic warning could be expected, such as an earthquake, terrorist attack or train 
derailment. The third category was for scenarios where significant warning was available, 

142 United States General Accounting Office, Content Analysis: A Methodology’ for Structuring and 
Analyzing Written Material (Washington, DC: U.S. General Accounting Office, 1996), 
http://www.gao.gOv/products/PEMD-10.3.l. 

143 Ibid. 
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scenarios such as hurricanes or winter storms that could be tracked for days or even 
weeks prior to impact. Everything else was considered a minimal warning event. Those 
which some amount of warning would be available like tornadoes or wild fires. These 
were listed as minimal warnings because there is a predictable season and while we 
cannot calculate where and when either will strike; there is some leverage of 
preparedness afforded. Specifically, they did not fall into either the No Warning or 
Significant Warning categories. The second category tracked the variety of hazards 
associated with the AARs. This would be used to detennine whether exercises were being 
conducted in accordance with real world events. The 14 categories ranged from events 
associated with criminal activity and terrorism to events caused by severe weather and 
natural hazards like ice storms and earthquakes. The hypothesis was that there would be 
inconsistencies with the frequency of events that actually occurred and those that served 
as an exercise scenario. 

The third and largest category was for recommendations. There were 104 separate 
classifications of recommendations taken from all of the AARs. These were 
recommendations from both exercises and real world events and were sorted 
independently. 

The last category was reserved for exercise objectives. These 71 objectives were 
taken from the exercise AARs and were coordinated for similarities among the 
recommendations in order to clearly associate the recommendations with the objectives 
when the analysis of the data was conducted. The thought here was that exercise 
objectives would not correlate to recommendations and would further the hypothesis that 
there is a need for the insertion of intelligence into the exercise design process. When 
reviewing the first category of AARs, the results (see Figure 5) show that exercises 
favored “no warning” scenarios over “significant warning” scenarios by a nine to one 
ratio (54% of exercises featured scenarios that were without warning while only 6% 
focused on scenarios with significant warning periods.) 

Real-world AARs showed almost an even distribution between “no warning” and 
“significant warning.” This is indicative of an inconsistency in exercise planning. If the 
emergency community responds more frequently to events where they have significant 
warning then should the exercises they train on be developed in a similar manner. 
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Looking at the scenarios that were extracted from the database of AARs, there are two 
noticeable variances (see Figure 6). Real-world events featured incidents of severe 
weather 42% of the time compared to only 25% as an exercise scenario marking a 71% 
increase from exercise to real world for that scenario. Similarly, terrorism was featured in 
23% of the exercises but only 6 % of the real world AARs. This represents a staggering 
262% increase from real world to exercises. 



No Minimal Significant 

Warning Warning warning 


■ Exercises ■ Real World Events 


Figure 5. Warning prior to Hazard 
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■ Exercises ■ Real World Events 

Figure 6. Scenario Distribution 


While terrorism-related scenarios may be popular their occurrence in real-world 
events was minimal and should be reflected proportionately in exercise scenarios. A more 
accurate use of the terrorist scenario is reflected if we look at exercises conducted after 
2008, though. In the AARs reviewed, prior to 2008, terrorism was used as an exercise 
scenario nine times as opposed to only three times since. This is consistent with its 
appearance in the real world AARs and is representative of the shift from preparedness 
for terrorism events to a more all-hazards approach in exercise planning. 144 Figure 7 is 
representative of the shift in AARs corresponding with the shift in preparedness scenarios 
as the data was analyzed pre and post 2008. 


I 44 Department of Homeland Security, Quadrennial Homeland Security Review Report. 
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■ Exercises ■ Real World events 
Figure 7. Terrorism Scenarios Pre- and Post-2008 
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■ Real World Events ■ Exercises 


Figure 8. Recommendations Sorted by Real-World Events 


When the recommendations were compared between exercises and real-world 
events the results were not surprising. Sorting the results by both real world events and 
exercises yielded similarities among needs. Among the top five in both categories were 
communications, training and exercise, problems with the Incident Command System 
(ICS), resource management and issues in the Emergency Operations Center (EOC). 
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Figure 8 lists the top 20 occurring recommendations among Real World AARs and 
Exercise AARs. 

4. Objective Recommendations 

The similarities among recommendations are the standard issues that the exercise 
design community is familiar with and these findings should come as no surprise. It 
should reason, then, that exercise objectives should show a similar concentration of 
issues. If we repeatedly expose the same areas for improvement in our AARs then these 
would likely be the areas that receive the most attention when it comes to our exercises. 
Figure 9 shows a somewhat different finding, however. Based on the content analysis 
conducted over the sample for this thesis, there were some significant differences. 

When looking at the top five based on disparity between recommendation and 
objective, resource management, documentation, multi jurisdictional issues, the public 
infonnation officer (PIO) and mutual aid and memorandums of understanding (MOUs), 
there is a noticeable divide between each. Documentation, for example, was an issue in 
48% of the real world AARs yet only found inclusion as an objective in 4% of exercises. 
Improperly documented associated costs are the primary reason local governments fail to 
receive reimbursement after a disaster. 145 So by addressing this one objective or, at a 
minimum, closing the gap, localities would be better able to qualify for reimbursement. 
More reimbursement would actually allow them to fund additional exercises. 


l 4 ^ State of Indiana, “Reimbursement Procedure for FEMA Public Assistance,” IN.gov, March 2006, 
http:// www. in. go v. 
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■ Exercise Objectives ■ Real World Recommendations 

Figure 9. Disparity between Real-World Recommendations and Exercise 

Objectives 


There were also recommendations that appeared frequently and were completely 
ignored as exercise objectives. Figure 10 demonstrates the recommendations that 
received no inclusion as objectives. This was a significant finding as the frequency of 
these recommendations was not minimal. 



■ Exercise Objectives ■ Real World Recommendations 


Figure 10. Recommendations Receiving Zero Inclusion as Objectives 
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D. ANALYSIS OF THE FINDINGS 

It is apparent from the analysis that recommendations from exercises and real 
world events are not being transferred into our exercise process. This is a problem. What 
this means is that the emergency management community is content to keep exercising 
the wrong issues, not exercising consistent problems and not learning from the 
intelligence gathered and analyzed after exercises and real world disasters. This 
intelligence should be the driving force to guide improvement and is being largely 
ignored. We seem to be carrying a predetermination toward our future exercises. A 
predetennination that suggests recommendations almost pre-scripted. 

E. LIMITATIONS OF THE METHOD 

It is understood that this method contains certain limitations. Listed below are the 
most obvious with an accompanying analysis and explanation. 

1. Sample Size 

The sample size of 101 AARs represents only a small portion of exercises and 
real world events. Sample size will always affect the data produced. 100 events will 
produce a more accurate pool of data than a sample of 10 but far less than a pool of 
1,000. The sample includes a mix of AARs from 30 states, which left out many locations. 
This was primarily due to the readily accessible nature of the chosen AARs. Specific 
AARs were not sought out with some exceptions. September 11 AARs were chosen as 
was the DC sniper AAR. The Oklahoma City bombing from 1995, the Columbine school 
shooting from 1999 were included as were select AARs from Hurricanes Katrina and 
Sandy in order not to neglect landmark events. 

2. Subjectivity in Selection 

One of the concerns when selecting the AARs for the data sample was the 
objectivity of the process. As explained earlier, AARs were chosen simply because of 
HSEEP compliance and U.S. origin. The intent was to maintain as much consistency in 
format and randomness in selection, though fully achieving this may not have been 
possible. 
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3. 


Distribution 


Several states were disproportionately represented in the sample and some states 
were not represented at all. While this was a result of the accessibility of the AARs and 
not a preference for any region, the unevenness of distribution may be incorrectly 
attributed to bias. 

4. Limitation Solutions 

The limitations explained above could be addressed by examining a larger sample 
of After Action Reports. While 104 may have served the purpose of this analysis, a larger 
cross section, including more locations, greater scenario inclusion and use of a wider 
sample of Federal, State and Local AARs, would have provided a more complex data set, 
likely resulting in a more accurate and complete analysis. 

Including more researchers would have given greater subjectivity to the analysis 
of the data. This, too, would have increased the depth of the results and reduced the 
possible appearance of analyst bias in the findings. 
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V. THE CONNECTION BETWEEN INTELLIGENCE AND 

EXERCISE 


It does not do to leave a live dragon out of your calculations, if you live 
near him. 

—J.R.R. Tolkien, The Hobbit 


A. INTELLIGENCE-FREE EXERCISE 

When exercises are conducted in a vacuum, that is, unaffected by surrounding 
environmental influences, they provide only of minimal benefit. Organizations train on 
what they are already efficient at and do very little to address or even identify their 
systemic gaps. It has been suggested that the planning process is more important than the 
procedures. Planning is often a result of the most recent disaster and not based on any 
systematic lessons learned. “This would pose no problem if the common sense notions 
and assumptions made about emergency time behavior were valid,” though social 
scientists have recently asked some serious questions about common expectations during 
disaster responses. 146 While there may be some predictability of a crowd during an 
emergency evacuation, properly preparing a population to respond during a disaster 
situation will raise the level of expected safety. 147 

When we evaluate exercises it is useful to use something similar to the Disaster 
and Exercise Perfonnance Information Collection Tool (DEPICT), developed by a joint 
civilian and military medical response team. 148 A repository for data collected from 
exercises or real world disasters, DEPICT allows field based observations to be broadly 
collected then analyzed with minimal bias. Frequently, the author of an evaluation is 
someone who was actually involved in the incident or was responsible for some aspect of 
the planning or response. It is not easy for one to impartially evaluate the actions of their 

l 4 ® Quarantelli, Human Resources and Organizational Behaviors .” 

1 47 Peter W. Blaich, The Benefits of Behavioral Research to the Fire Service-Human Behavior in 
Fires and Emergencies (Lincoln, NE: iUniverse, 2008). 

1 48 Smith et al., “Disaster and Exercise Performance Information Collection Tool.'’ 
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own organization. “Too often, post-disaster critiques turn out to be defenses or 
justifications of what was done, rather than objective assessments of problems and 
mistakes.” 149 By not incorporating an intelligence aspect into the planning of our 
exercises and thus our evaluations, we may be missing valuable lessons “because of the 
bias held for one’s own community or organization.” 150 

B. HSEEP FOR TERRORISTS 

How do terrorists leam? Can we assume that they have a structured framework 
for evaluation that is conducted after each attack, whether successful or not? By 
comparing a string of “failed” attacks we can sunnise a “lessons learned” structure of 
some sort. If so, then we as Homeland Security professionals should be able to further 
leam from this in an effort to be prepared for an evolving terrorist threat. As a terrorist 
group leams it can repair gaps in its structure and strengthen its capabilities. An ability to 
leam allows a terrorist group to adapt to its changing surroundings and goals by: 

• Developing, improving, and employing new weapons or tactics that can 
enable it to adjust its capabilities as necessary 

• Improving its members’ skills in applying current weapons or tactics 

• Collecting and utilizing the intelligence information needed to mount 
operations effectively 

• Thwarting countermeasures and improve its chance of surviving attempts 
to destroy it 

• Preserving the capabilities it has developed even if some of its members 
are lost. 757 

Observing the organizational learning that takes place among a group, more 
specifically than from individual to individual, we can begin to see areas where we can 
adapt our response in a similar, though far more positive manner. By understanding how 
terrorist groups continue to learn in an ever changing arena, response agencies can better 
organize to respond to and prepare for inevitable terrorist attacks. 


149 Auf der Heide, Disaster Response. 

150 Quarantelli, “Human Resources and Organizational Behaviors.” 

151 Jackson et at. Aptitude for Destruction. 
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In order to benefit from the organizational learning exhibited by a terrorist 
organization, it is necessary to understand why responders would want to learn from them 
in the first place. The objectives of the response organization need to be clearly 
understood. If the goal is preparedness and response, then understanding the terrorist 
intentions and tracking improvement plans, those tasked with preparing to respond to the 
next terrorist attack are anned with preparatory knowledge. 

The “Open Source Jihad” section of Inspire Magazine, described as “a resource 
manual for those who loathe tyrants ... a disaster for the repressive imperialistic nations: 
the open source jihad is America’s worst nightmare,” 152 provides easy-to-follow 
suggestions and instructions for would be, Home Grown Violent Extremists (HGVE). An 
article in the magazine posed this question: “Can I make an effective bomb that causes 
damage to the enemy from ingredients available in any kitchen in the world? The answer 
is yes.” And it went on to describe an easily assembled explosive device in the article 
“Make a Bomb in the Kitchen of Your Mom ,” 153 This was eventually the source for the 
explosive device used in the 2013 bombing at the Boston Marathon. 154 

This is significant both tactically and strategically. Tactically, it tells 
counterterrorism professionals to be stalwart in identifying the gathering of seemingly 
innocuous materials. Terrorists can create dangerous explosives easily, but more 
importantly, they can do so without leaving the country for expensive training that is 
available for security professionals to notice. 155 By understanding not only the methods 
but the motivation behind the terrorist intentions, we can begin to comprehend them 
strategically. 

It becomes possible to anticipate the terrorists’ next logical step by understanding 
their strategy. Understanding their particular goals and objectives and the tradeoffs they 


152 “A1 Qaida’s English-Language Terrorism Blueprint: The Investigative Project on Terrorism,” 
accessed February 18, 2014, http://www.investigativeproject.org/2056/al-qaidas-english-language- 
terrorism-blueprint. 

152 The AQ Chef, “Make a Bomb in the Kitchen of Your Mom,” Inspire, Summer 2010. 

154 Richard Spencer, “Boston Marathon Bombs: AlQaeda’s Inspire Magazine Taught Pressure 
Cooker Bomb-Making Techniques,” The Telegraph, April 16, 2013. 

155 “A1 Qaida’s English-Language Terrorism Blueprint: The Investigative Project on Terrorism.” 
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will be willing to make in order to accomplish these, allow counterterrorism professionals 
comprehend when available options have been minimized. 156 

Familiarity with the learning processes of terrorist groups could make it possible 
to develop countenneasures to combat or possibly take advantage of terrorist groups’ 
attempts to leam. Attacking the group’s “learning systems” may lessen its ability to 
evolve over time. 157 While the advances made by terrorist groups to adapt and leam are 
often obvious in hindsight, they are harder to identify proactively. If we are going to 
minimize their tactical advantage, however, early recognition of operational growth is 
critical. Intelligence analysts need to be able to extract valuable infonnation from the 
“noisy background of collected intelligence, and the analytical tools to discern what the 
infonnation means in terms of terrorist group adaptive activities.” 158 In order to make a 
calculated difference, analysts need to be able to detect terrorist groups’ efforts to change 
and adapt, anticipate whether those efforts will be successful, and act in order to limit 
terrorist groups’ ability to learn or undennine their learning efforts. 159 

In Aptitude for Destruction, Brian A. Jackson et al., characterize organizational 
learning as a four-part process composed of acquiring, interpreting, distributing, and 
storing information and knowledge. 160 Ah four parts must exist and be carried out 
successfully if knowledge is to become organizational—that is, clearly tied to group 
objectives, accessible to many different group members, and resistant to the loss of 
individual members. 161 This approach is similar to a corporate strategy where the system 
is designed to survive not because of the abilities of the personnel but in spite of them. 
The successes of the organization are procedurally based rather than personnel based. 
This is evident with a group like al-Qaeda (AQ), which exists in 2014 more as a 
philosophy than a hierarchical organization. This is not to say that al-Qaeda central 


156 Jackson et al., Aptitude for Destruction. Volume 2,1. 

157 Jackson et al., Aptitude for Destruction: Volume 1, xiv. 

158 Ibid. 

159 Ibid. 

160 Ibid. 

161 Ibid. 
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(AQC) no longer operates, but rather emphasize its many splinter organizations thriving 
under the AQ ideology. 162 


In August 2001, actor James Woods saw a number of apparently Middle Eastern 
men behaving so oddly on a flight from New York to San Francisco that he reported them 
to the FBI. This was possibly a test run for what would later be the September 11 high¬ 
jacking. 163 By monitoring routine actions by a transcontinental flight crew, the terrorists 
were able to gather significant information that may have provided the operational 
groundwork needed to carry out the most successful terrorist operation in history. The 
knowledge that terrorists conduct “dry runs” offers an insight into their meticulous 
planning and preparedness. 

1. Tactical Growth of Terrorist Organizations 

Understanding the tactical growth of two specific organizations may shed some 
light on how responders can react to an increasing threat as it develops rather than 
reacting to the aftennath of that process. 

a. The Radical Environmentalist Movement 

Groups such as the Environmental Liberation Front (ELF) and the Animal 
Liberation Front (ALF) have existed within the U.S. as leaderless organizations since the 
early nineties fighting for the rights of the environment and animals at the expense of 
hard working Americans. Since 1976, 1,100 criminal acts have been committed in the 
United States by radical environmentalist groups, resulting in more than $110 million in 
property damage alone, a figure that does not include the significant additional costs 
associated with lost research, increased security, and dampened productivity. 164 

Groups such as ELF and ALF share a unique organizational structure as they are 
essentially leaderless. The ELF considers any person who acts on behalf of the 


l® 2 Margaret Coker and Keith Johnson, “The Death of Osama Bin Laden As A1 Qaeda Adapts Its 
Tactics, Threat From Splinter Groups Persists,” Wall Street Journal, May 3, 2011. 

163 “James Woods Reported Suspicious Passengers to FBI” ABC News, accessed November 8, 2013, 
http://abcnews. go. com/Entertainment/story?id=l 02335. 

I® 4 Kristen Phillipkoski, “Eco-Terror Cited as Top Threat,” Wired, June 16, 2004. 
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organization, so long as such actions are consistent with ELF guidelines, to be an “Elf,” 
or agent of the organization, and any actions undertaken by Elves can be considered 
actions of the ELF as a whole. 165 This makes it difficult to track organizational learning 
and knowledge management among the group. 

The main aim is to base a group on so-called phantom cell networks or 
“autonomous leadership units” (ALUs) that operate completely 
independently of one another, but which are able, through the combined 
force of their actions, to precipitate a chain reaction that eventually leads 
to a national revolution. Integral to the concept is infonnation technology 
(IT), which is used to facilitate communication between like-minded 
individuals and as a means for circulating and distributing propaganda and 
information. Of particular importance are mediums such as the Internet, 
email and the world wide web. These cyber based mediums play a crucial 
role in ensuring that all individuals are kept fully abreast of events, 
allowing for planned responses and attacks but always on the basis of 
individual initiative. Used in this way, IT both overcomes the ‘tyranny of 
distance’ and obviates the necessity for orders and directives—thereby 
precluding the need or a physical group [or a centralized leadership], 166 

b. The Provisional Irish Republican Army 

The Provisional Irish Republican Anny (PIRA) has a 30-year history of 
organizational growth both strategically and tactically. 167 The group consistently fought 
to sustain a war of attrition against the occupying British with limited success. 168 One of 
the PIRA’s alleged leaders summarized the group’s strategy as “blattering on until the 
Brits leave.” 169 This caused a re-structuring of organizational goals. 
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“The remarkable evolution of this organization has been characterized by internal 
learning, an exceptional ability to adapt, reorganize, and restructure, and the impressive 
development of a highly efficient and multi-dimensional support apparatus” 170 

While the PIRA used Rocket-Propelled Grenades (RPGs) in many of its attacks, 
the weapon’s limitations caused the group to develop its own version, the Improvised 
Projected Grenade (IPG). 171 The newer weapons were more accurate and caused less 
accidents with unintended consequences; accidents which damaged sympathetic support. 

As the PIRA learned, it began operations aimed at specific targets which furthered 
its need for organizational improvements. As it increased its “out-of-theater” targets its 
intelligence needs also increased. This lead to strategic improvements as well. “In any 
conflict the tactical efficiency with which military operations are executed will be 
meaningless unless they form part of a coordinated plan to achieve political ends, because 
the success of a strategy can only be judged with reference to attainment of the overall 
political objective” 172 

c. Summary of Organizational Learning 

Both the PIRA and leaderless environmental groups like ELF and ALF have 
benefitted from organizational learning. There is no single way that organizations leam 
from past successes and failures. 173 Some professionals define organizational learning as 
a “change in behavior in response to stimulus,” 174 while others suggest learning is sought 
after, 175 a desire to improve based needs and shifting objectives. If an organization is 
going to continue to survive, it will have to benefit from a combination of techniques and 
frameworks. Homeland Security professionals will have to follow and anticipate the 
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organizational development of terrorist organizations and likewise, have to develop as 
organizations themselves. 


C. AGGRESSIVE INTELLIGENCE 

Is it possible, then, to use exercise design as an offensive weapon more so than a 
reactionary support tool. Can we, by combining the resources of an intelligence section 
with that of our exercise design capabilities, seek to prepare our responders and civilians 
for events that haven’t happened yet? The incorporation of competitive intelligence (Cl) 
into a cyclic exercise design process will allow an agency to monitor evolving threats that 
could potentially affect their agency or community and exercise accordingly. 

While the idea of Cl is primarily a corporate tactic, it allows a company to 
anticipate market developments proactively—rather than merely react to them. 176 The 
company is more able to remain competitive by improving its strategic decisions and 
leading to better performance against competitors. 177 An exercise design team, charged 
with the responsibility of preparing its agency for unforeseen disasters, maintains a 
similar need for a competitive edge. The competitor, in this case, is an evolving terrorist 
or complex natural threat, rather than a rival business. Anticipating the learning potential 
of an organization will help “predict” the next realistic threat and attempt to learn from 
that prior to its occurrence. 

D. THE MUMBAI—NAIROBI CONNECTION 

The 2008 terrorist attack in Mumbai can be used as an example of how 
intelligence could be used to create a predictive exercise. In November of 2008, 10 
terrorists attacked several locations in Mumbai, India killing at least 172 people. Referred 
to as “India’s 9/11,” the attack was carried out inexpensively and without the use of 
unconventional weapons. However, parts of this attack were noteworthy, specifically, its 
scope, complexity and the diversity of its targets. “The prolonged nature of the episode, 
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which went on for 60 hours with the steadily mounting death toll, made it a slow-motion 
shoot-out and siege that mesmerized the world’s news media.” 178 

According to the NYPD’s analysis of the attack at the Westgate Mall in Nairobi, 
Kenya, early on the afternoon of September 21, 2013, 

four terrorists associated with the Somali based terrorist organization Al- 
Shabaab launched an armed assault using rifles and hand grenades at the 
Westgate Shopping Mall in Nairobi, Kenya. The attack resulted in the 
deaths of 67 people and it took almost four days before authorities 
declared the scene safe. This terrorist attack was the largest loss of life in 
Kenya due to terrorism since the 1998 bombing of the U.S. Embassy in 
Nairobi in which 212 people were killed. This high profile armed assault 
successfully raised Al-Shabaab’s worldwide profile as a terrorist 
organization. It also clearly illustrates that anned assaults by terrorists on 
“soft” targets such as a shopping mall are a simple, effective and easy to 
copy tactic. 179 

Bruce Hoffman looked at Mumbai as a benchmark for terrorist success, stating, 
“the long, drawn-out assault in Mumbai produced round the clock coverage around the 
globe, something other terrorist groups want to emulate.” 180 This emulation appears to 
play out at Westgate. 

To summarize the two events, if we focus on the similarities between Mumbai 
and Nairobi, both events featured moving teams of attackers armed with assault weapons 
and grenades. Both targets were assumed to contain affluent visitors to the locations and 
it could be anticipated that most of the victims would likely only be familiar with the exit 
through which they entered minimizing their egress options. Both events lasted for days 
and were captured live for the world to see. Terrorism’s intent is to intimidate those 
watching. 181 “Terrorism is meant to hurt, not to destroy.” 182 The use of gunfire, as 
opposed to indiscriminate use of bombs as used previously in Mumbai attacks (2006 
Mumbai train attack, in which 209 people were killed by seven bombs, and the 1993 
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Mumbai attack where 257 persons died as a result of 13 blasts across the city), provided 
the “emotional value” sought by terrorists. 183 

There are similarities in the Westgate Mall. If, as a result of the mall shooting in 
Nairobi in 2013, intelligence analysts discover a systemic gap in evacuation policies in 
Nairobi, then that analysis will be shared with exercise design. A joint exercise between 
fire, emergency medical and law enforcement personnel as well as private local 
stakeholders with a vested interest in mass evacuation of customers will be developed, 
conducted and evaluated. The location is not as important as the similarities in 
consumers. For example, the mall in Nairobi contained a transient population, marginally 
familiar with the facility and not fully familiar with its exits or available areas of refuge. 
These factors influenced the chaotic movement of the crowd which fueled the attacker’s 
assault targets. 184 

If the development of the exercise scenario includes multiple insults to nonnal 
movement with various points of egress blocked, then the challenge of the exercise 
becomes compromised evacuation. The evacuation exercise becomes the benefit to the 
facility as a result of the intelligence provided. 

1. Lessons Learned from Mumbai and Nairobi 

If we were required to conduct an after action review of the two events from the 
point of view of the “bad guys,” what would be areas for improvement? Would the 
following recommendations be considered appropriate? 

• RECOMMENDATION: While the fire set at the Taj Hotel was effective 
in garnering media coverage for a prolonged period, it should have been 
set on a lower floor and set sooner . 185 This would ensure more trapped 
occupants and a higher causality count. 


183 Rabasa et at, The Lessons of Mumbai, 6. 

184 Daniel Howden in Nairobi, “Terror in Westgate Mall: The Full Story of the Attacks That 
Devastated Kenya,” The Guardian , October 4, 2013. 

183 Rabasa et al., The Lessons of Mumbai. 


60 



• RECOMMENDATION: Disruptive explosions away from the hotel 
caused confusion among responders. Explosions early in Nairobi had a 
similar confusing and chaotic effect . 186 

• RECOMMENDATION: Response of fire and police units is often 
predictable so the effectiveness of prepositioned IEDs should slow 
response. 

• RECOMMENDATION: Use of a “command post” allowed for 
coordinated communications as well as real time intelligence for attackers 
in both Mumbai and Nairobi. 757 

Another incident worthy of review, at this point, would be the Westfield Galleria 
Mall arson attack fonn October of 2010. During this Roseville, California, incident, an 
armed suspect, believed to be carrying an explosive device, entered a crowded, upscale 
mall. The suspect later started a fire and barricaded himself in the back of one of the 
stores. Through the confusion, the fire sprinklers were deactivated and were not 
reactivated until 71 minutes into the incident. 188 While it is unclear as to why the 
sprinklers were deactivated, what is clear is how this exacerbated the fire. The important 
lesson learned here, for the would-be terrorist is that the absence of the fire suppression 
system, for over an hour, resulted in 55 million dollars in damage to the mall. 189 One of 
the goals of terrorism is financial disruption to the target. The motivation of the attacker 
in Roseville, for the purposes of this evaluation, is irrelevant. What is relevant is that one 
armed attacker occupied a mall, terrorized its occupants and set fire to the facility, 
holding responders at bay. His actions caused excessive damage. 

E. INTELLIGENT EXERCISE THEORY 

1. Predictive Scenario 

After an emergency, such a focus on the most recent event is common and the 
planners often concentrate exclusively on planning for future scenarios of the same 
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type. 190 Ideally, intelligence analysts should focus on what the terrorists view as their 
lessons learned and provide this to exercise planners. This allows for the development of 
a predictive exercise where we begin to prepare for what may be next. 

If terrorists in the U.S. were intent on carrying out an active shooter event, then 
the lessons learned from Mumbai and Nairobi should help in the planning stage. 
Supplementing this intelligence with what could, conceivably, be learned from Roseville, 
should yield a complex active shooter event, carried out by four to six gunmen in a 
reasonably crowded facility e.g., a mall, theater or rail station, occupied by transient 
population unfamiliar with all the exits or nuances of the building. 

By triggering several explosions prior to entering the facility, the attackers could 
cause a distraction. Then, by compromising the fire suppression system the attackers 
could begin moving through the facility firing their weapons, eventually starting the fire 
and either trapping victims on floors above or siphoning them to specific exits using the 
directionality of the fire. This corralling would leave the victims extremely vulnerable. 

If this event were to take place in a location with an extremely vulnerable 
population, a hospital for example, then the casualty count could be significant. As would 
the emotional impact the attack would have on a U.S. population. A New York hospital 
emergency planner, speaking on the condition of anonymity, stated her biggest concern in 
preparing the hospital was its over 50 exits and the hospital’s inability to secure them all. 

The incorporation of intelligence allows us to anticipate what challenges may lay 
ahead and begin to prepare for them. 

2. Predictive Exercise 

Taking the lessons learned from previous events and adding them to what we can 
expect would be an attractive soft target in the U.S., should lead us to conduct evacuation 
exercises that focus on compromised evacuation routes. As people unfamiliar with 
additional means of egress, other than the one through which they entered, can be 
expected to behave as part of a crowd during unscheduled evacuation, we, as emergency 
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planners, need to provide scenario based exercises that would help a facility develop 
alternative response plans when faced with a chaotic, aggressive obstacle. 

When developing an exercise intended to prepare people for any emergency, the 
scenario is not nearly as important as are the objectives. Using the intelligence from 
above, it is important to realize that our exercises do not need to mirror our intelligence, 
merely shadow the objectives. What this means is that it is not necessary to conduct an 
active shooter exercise in a hospital to provide hospital personnel with the benefits of the 
intelligence shared. A Tabletop Exercise (TTX) conducted in a hospital with discussions 
considering a variety of compromises to safety should lead to anticipatory solutions. 

3. Intelligent Exercise Cycle 
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Figure 11. At Left is the Exercise Cycle, at Right, the Intelligence Cycle 

In order for exercises to remain current and relevant, they need to be based on 
new intelligence. Ideally, this will provide the exercise cycle with constant input. The 
cycle, in this way, however, is stagnant within the exercise agency. As the new 
intelligence is fed to the exercise cycle, the exercises are conducted and evaluated, 
improvement plans are developed as are corrective action plans. New exercises are 
created and the cycle continues. As the intelligence community gains new information, 
this is analyzed and processed then disseminated according to the needs of the consumer. 
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Feedback is received from within the intelligence community and the cycle continues as 
in Figure. These two cycles can benefit from each other through their mutual cooperation. 
The development of the Intelligent Exercise Model (IEM) will consist of integrated 
versions of both the exercise design cycle 191 and the intelligence cycle. 192 Each cycle 
operates independently while running a constant flow of feedback between the two 
entities. As evaluation infonnation from one cycle is fed into the other, each cycle 
supports the momentum of the other, theoretically leaving the cycles in perpetual motion 
dependent upon influences from each other. The key to the sustainability of the procedure 
is that the entire process is constantly fueled by new intelligence. The feedback from the 
exercise design team is delivered, post-exercise, to the intelligence producers. They, in 
turn, continue to provide intelligence to the exercise design professionals in an effort to 
stay infonned concerning current trends and threats. 
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Figure 12. Intelligent Exercise Model 


The research conducted for the analysis of this thesis produced the Intelligent 
Exercise Theory (IET). Which can be defined as “The use of intelligence gained from a 
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disaster or terrorist attack to prepare for future events, while only recreating the most 
salient facets of the event to drive the exercise design learning cycle in anticipation of 
future crises. This is to suggest that reconstructing the entire disaster for the purposes of 
exercise is not nearly as effective as exercising those aspects of the disaster deemed 
significantly influential as to manipulate reaction to future disasters. ” 

Exercises are often location driven. This means that if a facility of interest 
becomes available, the calendar turns out to be the motivating factor. Projects are derived 
from relationships that exist between agencies and a mutual interest in a particular facility 
rather than based on a realistic threat assessment of locations. This limits the scope of the 
exercise. By narrowly focusing our scenarios based on the availability of exercise 
partners, rather than allowing current threats and vulnerabilities to steer our exercise 
goals and objectives, we are limiting opportunities for conducting successful, effective 
exercises. The union of intelligence and exercise design into a symbiotic process will 
simultaneously broaden the inclusion of exercises while pinpointing the focus of design 
capabilities. 



Figure 13. Technology Acceptance Model 193 


Since the IEM integrates variations of two accepted models; the intelligence 
cycle 194 and the HSEEP 195 exercise cycle, participants from exercise and intelligence are 
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already familiar with them, minimizing the learning curve. The fact that these two models 
are already widely accepted means that there should already exist an anticipated ease of 
use as well as an expectation of value added to the process. 

It is important to understand the details of objectives. We can’t find a solution 
until we have understood the problem. Gathering infonnation from many sources will 
broaden the field of information and eventually improve the focus of the project; what the 
final project is supposed to look like. Initially, an increase in infonnation leads to better 
solutions, but there comes a point at which more infonnation has a diminishing effect. 
Infonnation by itself does not produce solutions. We need the ability to work with the 
information effectively. 196 The incorporation of intelligence into exercise design provides 
intelligence producers and analysts with a measureable consumer. While intelligence 
often is left with intangible perfonnance successes, the consumer and producer rarely 
exchange thoughts. 197 The development of exercises based directly on intelligence 
delivered can provide a performance measure and apply realistic objectives, as well as 
collaborative feedback based on new intelligence and proposed exercises. The 
intelligence officers actually contribute and participate in the next step in the intelligence 
cycle. 

The exercise evaluation will then result in a comprehensive After Action Report 
and Improvement Plan (AAR/IP) that will be disseminated among responding agencies 
and the private stakeholders. Developed cooperatively by responders and private 
industry, the AAR/IP can benefit all agencies involved in the exercise. Lessons learned 
from these exercise should be shared among participating agencies but, more importantly, 
made available to non-participating agencies that would benefit from the findings. 
Competitive interests between competing agencies (i.e., hotels, hospitals, office 
buildings) need to be set aside and allow the greater interest of community resilience take 
precedence. Each individual agency or stakeholder needs to view itself first as a member 
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of the community and secondly as an individual interest. This eliminates multiple in¬ 
groups focused on their own needs and creates a collective in-group focused on the 
community’s needs. 

This is the idea currently in use in Israel 198 where the community is focused on 
the protection of Israeli interests ahead of individual interests. The Israeli model is “based 
on a series of concentric circles of increasing protection, with the individual citizen at the 
center.” 199 This is a definitive, almost literal view of community resilience, where the 
community has an active participatory role in its own security. To provide the civilians 
with the necessary tools to respond appropriately and immediately when faced with an 
emergency, the Israel Defense Forces’ Home Front Command (IDF HFC) orchestrates 
exercise and drills that prepare the civilian population to interact, where necessary with 
first responders. 

Community resilience is not just about how we respond during times of crisis. 
Additionally, it raises questions that challenge how we build strong and sustainable 
communities in the face of the demographic, social, and economic changes of the 21 st 
century. For example, how do we bring together government and private partners 
equitably in planning discussions? How do communities care for their most 
vulnerable? 200 By exercising and training together, each agency begins to understand the 
capabilities and needs of the other in an effort to support the overall vision of Israeli 
security. This is, simply stated, the tenet of community resiliency. 201 

4. New Solutions Create New Problems—A Case Study 

In 2008, the New York City Fire Department’s (FDNY) Exercise Design team 
created a bus bombing scenario that allowed over 100 FDNY responders to exercise in a 
realistic atmosphere, complete with an eviscerated bus, numerous simulated injuries and 
secondary threats. This exercise, consistent with global events and threats, i.e., the 2005 
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terrorist attack on London’s mass transit system, was later incorporated into the FDNY’s 
regular training schedule and allowed every member of the Department, fire and medical, 
to participate. In May of 2010, members of the FDNY responded to an attempted car 
bomb in Times Square and acted in accordance with their training to safely mitigate a 
potential disaster. 202 This cycle, consisting of: 

Intelligence —*■ Exercise Design —> Training —> Practical Application 

should serve as the model for Intelligence-led Exercise design. The events of May 2010 
actually validate the IEM. Consistent, determined training provides a path for effective 
responder perfonnance. 

Any new procedure runs the risk of failure. To incorporate a cooperative working 
relationship between an intelligence section and its agency’s exercise design personnel is 
going to run somewhat contradictory to current operating procedures. Intelligence is now 
collected, analyzed and evaluated by the intelligence section. Intelligence is then 
packaged for dissemination. The IEM will need at least a year to test for effectiveness. If 
ineffective, the transition back will also take time. This could, conceivably, result in two 
year absence from the traditional established exercise design procedures that a skilled 
team has grown familiar with. If affected personnel, reluctant to change initially, are now 
faced with reinstituting the methods they had fought to keep, the possibility exists of 
never re-establishing the original procedures. 

The new integration may also develop new problems not yet conceived. In order 
to maintain smooth effectiveness, these will have to be identified and controlled on an 
individual basis. 
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VI. CONCLUSION 


A. SUMMARY OF ISSUES 

The intent of this thesis was to explain the need for incorporated intelligence into 
an agency’s exercise design model. The idea of exercise design is to validate existing 
training. The addition of intelligence into the planning process means that we will be 
preparing to meet tomorrow’s challenges as well as yesterdays. This allows us to think 
forwardly and attempt to predict where our future challenges lie, rather than solving 
problems where we have already demonstrated measured success. 

The fusion of intelligence and exercise design will produce a more integrated 
exercise product that the FDNY can share the benefits with other city agencies as well as 
private sector entities. The goal is not simply to improve the exercise design methods but 
to increase the usability of exercise design as a building block of community resiliency. 
The incorporation of intelligence with preparedness efforts is essential in order for the 
community to achieve true resiliency. Improving public and private communications and 
the exchange of infonnation is possible through the strengthening of these relationships. 
The new efficient development of exercise design should highlight exercise deficiencies 
overlapping the concerns of city agencies and private industry and lead to the creation of 
exercises that produce more value that can be reinserted into the exercise design model 
and continue to build on the cycle. The fusion of exercise design with intelligence will 
perpetuate both the exercise design and intelligence cycles, resulting in the development 
of predictive exercises that build a closer public-private relationship and a more resilient 
community. 

The plans listed above include a recalibration of the way exercises are planned 
and conducted; from a stagnant, scenario driven enterprise to one fueled by current 
analytical expertise. 

B. FINAL THOUGHTS 

If we wish to have collaboration between agencies, we must first foster an 

atmosphere whereby mutual acceptance of ideas exists. By using intelligence analyzed 
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from real world events, predictive exercises can be developed incorporating the needs of 
the community, public and private. In order to implement this strategy to benefit a 
department like the FDNY and a city like New York will require the creation and staffing 
of additional positions. 

Initially, this is important to both the FDNY’s Center for Terrorism and Disaster 
Preparedness’s (CTDP) Intelligence division and Exercise design section. The success 
will have a cascading effect. As the CTDP conducts more intelligence lead exercises, 
those involved in the exercises, the FDNY personnel and community stakeholders, will 
begin to enjoy more realistic results in the exercise. This can be measured in increased 
collaboration in future exercises as well as increased sharing of intelligence between the 
agency and private stakeholders. 

The idea of intelligence driven exercise design may seem a simple and logical 
one. We receive information from various sources every day. This information is 
analyzed and compared to what is already known, then is synthesized and distributed as 
intelligence. A careful analysis of this intelligence should enable us to develop exercises 
that look forward rather than backward, to allow responders to learn from yesterday’s 
emergencies and prepare for tomorrow’s. 

C. RECOMMENDATIONS 

While the core of this thesis was directed toward the exercise design and 
intelligence communities, the recommendations will be directed primarily toward the 
FDNY and its intelligence and exercise needs. In order to successfully implement the 
intelligent exercise model, the following recommendations are suggested. 

1. Promote Intelligent Exercise Theory 

As this thesis has made clear, Intelligent Exercise needs to be the future of 
exercise; and the future of intelligence. The two communities need to realize their 
significance to one another and work on developing mutual objectives. As each realizes 
the other is part of their in-group and that they are not individual entities competing for 
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the same commodity but partners working toward the same goals, they can begin to 
develop intelligent exercises that benefit the entire community. 

2. Clearly Define the Exercise Design Model 

Intelligent exercise is a simple fusion of the intelligence cycle and the HSEEP 
exercise design cycle, the process needs to be fully explained. The benefits that come 
from the intersection of the cycles at the evaluation process drive the incorporation of 
intelligence into exercise. The re-evaluation process will have to be apparent as will an 
anticipated benefit and an obvious ease of use. 

3. Create an Exercise Intelligence Liaison Officer 

This position would serve as a liaison between exercise design and intelligence. 
Not dedicated to either exercise design or intelligence, but committed to the fusion of 
both interests, the liaison position will function as a bridge between current trends and 
future exercises. By understanding the principals of intelligence gathering and 
infonnation sharing as well as the fundamentals of exercise design, the officer assigned to 
this task can link the two fields in an effort to increase the value of preparedness 
exercises. It would also serve the fusion of the two sections to obtain a level of security 
clearance for this officer so he may further bridge the gap between the fields. The cost 
will be dependent on the member filling the new position. This can be satisfied by 
assigning a current member of either the intelligence section familiar with exercise design 
or similarly with a member of the exercise design team possessing intelligence 
experience. Since both sections are already established, the impact on cost should be 
minimal. The cost can be split across the budgets already allocated for the two sections. 
However, transitioning a member between both sections will essentially leave both 
sections with a partial personnel deficit as the new position will remove them from 
current responsibilities. Conversely, if a new member needs to be assigned from outside 
the Center, the cost will be consistent with the member’s pay rate and will require new 
budget lines. The creation of a new position will definitely have an impact on the grant 
funded budget that the CTDP survives on. 
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This initial financial influence can be offset by future anticipated efficiency. By 
increasing the value of exercises conducted and relying on participation from private 
stakeholders, the quantity of exercises conducted should decrease while the value per 
exercise increases. It can be anticipated that the cost of fulfilling the new Liaison position 
will be, at least partially, compensated through the efficiency of the new procedure. 

The implementation procedure should not be lengthy but identifying the new 
exercise intelligence liaison officer may take several months. The position is new and 
untested. Identifying an appropriate officer who is a Master Exercise Practitioner (MEP) 
and an experienced and qualified Intelligence analyst will be time consuming. A person 
with these specific characteristics may not exist and the position may be filled by an 
individual with only partial qualifications. Also, once filled, as the position is un¬ 
established, there will be a positional learning curve. The size of the learning curve will 
be dependent on individual skills. 

4. Develop a N.Y. City-Centric Exercise Design Platform 

Similar to the DHS National Lessons Learned Information Sharing (LLIS), 203 this 
online platform would actively seek collaboration citywide when intelligence dictates the 
need for a specific exercise. Made up of public and private stakeholders, agencies would 
be responsible to share intelligence and ideas and offer personnel committed to the design 
team. At the conclusion of the exercise, collaboration on the AAR would result in a 
shared NYC Lessons Learned (NYCLL) platform that would serve as a foundation for 
future collaborative exercises. Transfer of knowledge is not an easily identified 
process, 204 but the potential return on investment that can be associated with collaborated 
exercises can be assumed to be significant. When multiple agencies look at an issue 
through a variety of perspectives we can gain greater insight into potential areas of 
concern invisible to one agency or another. 

As a result of its commitment to improvement through exercise, the FDNY 
currently employs over 40 Master Exercise Practitioners (MEP). Many NY city agencies 
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have zero. That doesn’t mean they do not have an interest in exercise, they may not have 
the personnel or resources available. This limits their capability to contribute. By making 
lessons learned and AARs available on an open source, citywide platform, each agency 
would have the potential to leam from every exercise and all city agencies. 

As a result of the NYCLL, agencies without exercise design personnel could also 
find more openness in the NYC exercise design community allowing them to observe 
and/or contribute to exercises from other agencies and remain within their budgetary or 
personnel constraints. 

New ideas drive change. Change, however, is often met with resistance. This 
resistance is what we need to overcome. The idea of moving procedures forward, while 
on the surface would seem to resonate progress, is frightening or intimidating to some. 
Likewise, interagency biases may prevent collaboration between agencies, public and 
private. While many agencies are comfortable exposing deficiencies internally, they tend 
to be less comfortable having these publically known. 205 This can be true internally as 
well as externally. If each section views itself as part of the in-group and the other section 
as the out-group, then collaboration will not flourish. 206 By developing exercises based 
on intelligence together, both sections will realize that they are part of the same in-group. 
Only then can true effective collaboration be expected to grow. 

D. METRICS OF SUCCESS 

Stakeholders will look to participate in the development of these exercises, as 
FDNY exercise design improves its reputation among the city exercise design 
community. Insurance underwriters, aware of the benefits reaped by their clients will 
encourage this participation by offering incentives to these stakeholders. Therefore, the 
stakeholders, private industry, can offset the cost of the exercise by partially funding this 
exercise based on the cost savings rewarded them by their insurance companies. 
Consequently, by improving the exercise design product and exploiting its collaborative 
benefits citywide, the new process can become self funding. 

205 Best, Intelligence Information. 

206 Tajfel, Social Identity and Intergroup Relations. 
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In order to determine successfulness, progress will have to be measured 
incrementally. First, the length of time to identify an Intelligence exercise liaison should 
serve as a significant indicator of anticipated success. A prolonged search with several 
starts and stops could be indicative of future systemic problems. Conversely, if the 
position is fdled quickly and the transition begins cooperatively, expectations of success 
are realistic. The positional requirements are only part of the anticipated systemic 
concerns. 

The true measure of success will be the observation of increased collaborative 
interest among additional city agencies (e.g. NYPD) and private stakeholders with FDNY 
exercise design. It is possible that knowledge developed from the analytic preparedness 
activities may be transferred to individuals that participated in the exercises by 
discussions or other activities not necessarily acknowledged by the individuals. 207 This 
means that some of the benefits derived during the exercises may simply result in 
performance improvements with individual applications. Employees and responders may 
find themselves more prepared and familiar with their evacuation and response 
capabilities in ways that are not tangibly measureable, but result in overall improvements 
that extend to non-emergency situations. 

E. POTENTIAL OPPOSITION 

There is an organizational reluctance to change for several reasons. Teemu 
Reiman, and Carl Rollenhagen outline human and organizational biases that have an 
effect on the management of safety in four thematic areas: beliefs about human behavior, 
beliefs about organizations, beliefs about infonnation and safety models. 208 It is possible 
that the new procedure will be met with resistance from both the exercise design and 
intelligence sections. The fusion of exercise and intelligence is not the way things are 
currently done and siloed operations are often preferred to change. The two divisions 
currently operate with stovepipe functionality. That is to say, while they are certainly 
aware of each other, there exists a territorial protection around which each section 
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operates and cooperative analysis rarely occurs. The inclusion of intelligence into 
exercise may result in mission creep, where exercise design feels that intelligence is 
guiding their exercise plans. Likewise, Intelligence may feel as if Exercise is trying to 
coax intelligence to fit the desired exercise plan. The two sections will have to realize a 
common goal and work together to achieve it. Each will also have to respect the role of 
the other in order to maintain unified objectives. In other words, each section will 
function as independent elements of the same corporation realizing the same goals. In 
order to achieve this, an exercise can be developed with the interests of the intelligence 
section featured prominently. By including the intelligence personnel in each step of the 
exercise process, they can see, firsthand, the benefits that will be gleaned from this 
fusion. Allowing them to have structural input toward the completed exercise will let 
them direct the exercise to include the intelligence they have analyzed including 
anticipated actions by players and antagonists, whether man-made or natural. 

Intelligence has historically been labeled as a trade that shares information 
reluctantly. 209 Exercise design needs to establish itself as a legitimate consumer of 
intelligence in order to encourage intelligence sharing. To accomplish this, exercise 
design is going to have to demonstrate a value added to the intelligence community. The 
incorporation of the two cycles demonstrates this. The evaluation step in each cycle is 
designed to support the movement of the other. Exercise design can provide valuable 
action items recovered after an exercise showing a need for additional intelligence. 

After Action Reports written at the conclusion of an exercise can show successes 
from intelligence inserted into the process but, more importantly, exhibit a need for 
additional intelligence. 

This model is asking the FDNY intelligence community to share openly. Not only 
to share intelligence but to offer analysis relevant to the exercise design community. The 
fact that they are being asked to share internally may not matter. This transition from two 
independent sections to two parts of the same whole operating cohesively may meet some 
opposition. 
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Specialized training is required to become an Intelligence analyst. Exercise design 
professionals receive training to become Master Exercise Practitioners (MEPs). Both 
specialists are proud of their certifications and protective of them. If the model is 
successfully implemented within the FDNY, then it could serve as the ideal for 
incorporation between the two communities beyond the Department. This could create 
territorial defensiveness by both communities, if either the intelligence or exercise design 
community feels the other is attempting to assume their responsibilities without the 
benefit of appropriate training. 

F. ADDITIONAL RESEARCH NEEDED 

Several areas require additional research in the area of Intelligent Exercise. The 
relationship between recommendations and objectives needs to be expanded. While the 
research conducted was sufficient for this study, it should be continually expanded and 
monitored. This will produce evidence of improvement trends or show continued 
stagnation in areas that require improvements. 

The Intelligent Exercise Theory and the Intelligent Exercise Model need to be 
exercised. A viable metric must be developed by which the theory can be tested and 
evaluated and resources and time committed to its use and necessary improvement. To 
simply adapt a new policy without a willingness to support its progress is 
counterproductive to the theory’s premise. 

Collaboration should be part of every agency’s strategic plan. Public and private 
agencies must perceive themselves as part of the larger community first and foremost and 
realize that their participation in an agency should come secondary. Community 
resiliency should be the goal of every member of the community. 
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SUPPLEMENTAL 


Analysis of 101 After Action Reports Research conducted for this thesis was 
based on 101 After Action Reports (AAR) that were evaluated for themes in 
recommendations and objectives. The AARs were separated by real-world events and 
exercises. Each exercise AAR was coded for inclusion into 71 categories of objectives 
and 107 categories of recommendations. Each real-world AAR was coded for 
recommendations in the same categories. The frequencies of occurrence were then 
compared and analyzed. 

Those interested in obtaining the supplemental can contact Dudley Knox Library 
at Naval Postgraduate School. 
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